2022-09-27T15:24:18.5586518Z Requested labels: linux.2xlarge 2022-09-27T15:24:18.5586639Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/85462/merge 2022-09-27T15:24:18.5586669Z Waiting for a runner to pick up this job... 2022-09-27T15:24:22.5108371Z Job is about to start running on the runner: i-00a9b5b3060f0b9ae (organization) 2022-09-27T15:24:25.6571956Z Current runner version: '2.296.2' 2022-09-27T15:24:25.6577575Z Runner name: 'i-00a9b5b3060f0b9ae' 2022-09-27T15:24:25.6578093Z Runner group name: 'Default' 2022-09-27T15:24:25.6578719Z Machine name: 'ip-10-0-3-234' 2022-09-27T15:24:25.6580580Z ##[group]GITHUB_TOKEN Permissions 2022-09-27T15:24:25.6581253Z Actions: read 2022-09-27T15:24:25.6581545Z Checks: read 2022-09-27T15:24:25.6581838Z Contents: read 2022-09-27T15:24:25.6582088Z Deployments: read 2022-09-27T15:24:25.6582413Z Discussions: read 2022-09-27T15:24:25.6582706Z Issues: read 2022-09-27T15:24:25.6582950Z Metadata: read 2022-09-27T15:24:25.6583234Z Packages: read 2022-09-27T15:24:25.6583563Z Pages: read 2022-09-27T15:24:25.6583832Z PullRequests: read 2022-09-27T15:24:25.6584173Z RepositoryProjects: read 2022-09-27T15:24:25.6584496Z SecurityEvents: read 2022-09-27T15:24:25.6584749Z Statuses: read 2022-09-27T15:24:25.6585031Z ##[endgroup] 2022-09-27T15:24:25.6588118Z Secret source: None 2022-09-27T15:24:25.6588665Z Prepare workflow directory 2022-09-27T15:24:25.8734017Z Prepare all required actions 2022-09-27T15:24:25.8893024Z Getting action download info 2022-09-27T15:24:26.1347169Z Download action repository 'pytorch/pytorch@master' (SHA:01dbbeeeb5ab7ede28e333982e98713282a0e4b8) 2022-09-27T15:24:28.9263298Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767) 2022-09-27T15:24:29.0528714Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:3c1d75049465d7dfa70acca6d80b9c5c06ff4886) 2022-09-27T15:24:29.2808140Z Getting action download info 2022-09-27T15:24:29.4258417Z Download action repository 'malfet/checkout@silent-checkout' (SHA:f63e9e15406be6060f159846cd2e098f759c5246) 2022-09-27T15:24:29.6045145Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml 2022-09-27T15:24:29.6046783Z ##[group] Inputs 2022-09-27T15:24:29.6047069Z build-environment: linux-focal-py3.7-gcc7 2022-09-27T15:24:29.6047847Z test-matrix: { include: [ { config: "default", shard: 1, num_shards: 2, runner: "linux.2xlarge" }, { config: "default", shard: 2, num_shards: 2, runner: "linux.2xlarge" }, { config: "distributed", shard: 1, num_shards: 2, runner: "linux.2xlarge" }, { config: "distributed", shard: 2, num_shards: 2, runner: "linux.2xlarge" }, { config: "functorch", shard: 1, num_shards: 1, runner: "linux.2xlarge" }, { config: "docs_test", shard: 1, num_shards: 1, runner: "linux.2xlarge" }, { config: "jit_legacy", shard: 1, num_shards: 1, runner: "linux.2xlarge" }, { config: "backwards_compat", shard: 1, num_shards: 1, runner: "linux.2xlarge" }, ]} 2022-09-27T15:24:29.6048700Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29 2022-09-27T15:24:29.6049038Z sync-tag: 2022-09-27T15:24:29.6049210Z ##[endgroup] 2022-09-27T15:24:29.6049657Z Complete job name: linux-focal-py3.7-gcc7 / test (distributed, 1, 2, linux.2xlarge) 2022-09-27T15:24:29.6712115Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master 2022-09-27T15:24:29.6712441Z with: 2022-09-27T15:24:29.6712630Z submodules: recursive 2022-09-27T15:24:29.6712816Z fetch-depth: 0 2022-09-27T15:24:29.6712985Z env: 2022-09-27T15:24:29.6713165Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:24:29.6713346Z ##[endgroup] 2022-09-27T15:24:29.6929392Z ##[group]Run retry () { 2022-09-27T15:24:29.6929657Z retry () { 2022-09-27T15:24:29.6929882Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2022-09-27T15:24:29.6930094Z } 2022-09-27T15:24:29.6930266Z echo "${GITHUB_WORKSPACE}" 2022-09-27T15:24:29.6930483Z if [ -z "${NO_SUDO}" ]; then 2022-09-27T15:24:29.6930707Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2022-09-27T15:24:29.6930908Z else 2022-09-27T15:24:29.6931090Z  retry rm -rf "${GITHUB_WORKSPACE}" 2022-09-27T15:24:29.6931280Z fi 2022-09-27T15:24:29.6931504Z mkdir "${GITHUB_WORKSPACE}" 2022-09-27T15:24:29.6947103Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T15:24:29.6947342Z env: 2022-09-27T15:24:29.6947518Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:24:29.6947687Z NO_SUDO: 2022-09-27T15:24:29.6947853Z ##[endgroup] 2022-09-27T15:24:29.7125236Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-09-27T15:24:31.4472368Z ##[group]Run malfet/checkout@silent-checkout 2022-09-27T15:24:31.4472605Z with: 2022-09-27T15:24:31.4472807Z ref: 52424e2bf38e454d535881fed9628d3e20f4f944 2022-09-27T15:24:31.4473016Z fetch-depth: 0 2022-09-27T15:24:31.4473188Z submodules: recursive 2022-09-27T15:24:31.4473376Z quiet-checkout: true 2022-09-27T15:24:31.4473579Z repository: pytorch/pytorch 2022-09-27T15:24:31.4473955Z token: *** 2022-09-27T15:24:31.4474122Z ssh-strict: true 2022-09-27T15:24:31.4474318Z persist-credentials: true 2022-09-27T15:24:31.4474636Z clean: true 2022-09-27T15:24:31.4474826Z lfs: false 2022-09-27T15:24:31.4475011Z set-safe-directory: true 2022-09-27T15:24:31.4475197Z env: 2022-09-27T15:24:31.4475355Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:24:31.4475539Z ##[endgroup] 2022-09-27T15:24:31.5590818Z Syncing repository: pytorch/pytorch 2022-09-27T15:24:31.5592412Z ##[group]Getting Git version info 2022-09-27T15:24:31.5592834Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2022-09-27T15:24:31.5593284Z [command]/usr/bin/git version 2022-09-27T15:24:31.5593471Z git version 2.37.1 2022-09-27T15:24:31.5599116Z ##[endgroup] 2022-09-27T15:24:31.5612458Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/ab502a80-6690-4793-9a53-0bf13949bfcd' before making global git config changes 2022-09-27T15:24:31.5612976Z Adding repository directory to the temporary git global config as a safe directory 2022-09-27T15:24:31.5618391Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-09-27T15:24:31.5655503Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2022-09-27T15:24:31.5658874Z ##[group]Initializing the repository 2022-09-27T15:24:31.5663268Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-09-27T15:24:31.5691128Z hint: Using 'master' as the name for the initial branch. This default branch name 2022-09-27T15:24:31.5691697Z hint: is subject to change. To configure the initial branch name to use in all 2022-09-27T15:24:31.5692270Z hint: of your new repositories, which will suppress this warning, call: 2022-09-27T15:24:31.5692661Z hint: 2022-09-27T15:24:31.5693167Z hint: git config --global init.defaultBranch 2022-09-27T15:24:31.5693686Z hint: 2022-09-27T15:24:31.5694086Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2022-09-27T15:24:31.5694723Z hint: 'development'. The just-created branch can be renamed via this command: 2022-09-27T15:24:31.5695118Z hint: 2022-09-27T15:24:31.5695472Z hint: git branch -m 2022-09-27T15:24:31.5695837Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2022-09-27T15:24:31.5701564Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2022-09-27T15:24:31.5729292Z ##[endgroup] 2022-09-27T15:24:31.5729851Z ##[group]Disabling automatic garbage collection 2022-09-27T15:24:31.5733835Z [command]/usr/bin/git config --local gc.auto 0 2022-09-27T15:24:31.5759012Z ##[endgroup] 2022-09-27T15:24:31.5759561Z ##[group]Setting up auth 2022-09-27T15:24:31.5766083Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-09-27T15:24:31.5795299Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-09-27T15:24:31.6040265Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-09-27T15:24:31.6070879Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-09-27T15:24:31.6320972Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-09-27T15:24:31.6362692Z ##[endgroup] 2022-09-27T15:24:31.6363164Z ##[group]Fetching the repository 2022-09-27T15:24:31.6368792Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2022-09-27T15:25:46.5107797Z [command]/usr/bin/git rev-parse --verify --quiet 52424e2bf38e454d535881fed9628d3e20f4f944^{object} 2022-09-27T15:25:46.5142479Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --quiet --no-recurse-submodules origin 52424e2bf38e454d535881fed9628d3e20f4f944 2022-09-27T15:25:47.6898155Z ##[endgroup] 2022-09-27T15:25:47.6898767Z ##[group]Determining the checkout info 2022-09-27T15:25:47.6899838Z ##[endgroup] 2022-09-27T15:25:47.6900292Z ##[group]Checking out the ref 2022-09-27T15:25:47.6905342Z [command]/usr/bin/git checkout --quiet --force 52424e2bf38e454d535881fed9628d3e20f4f944 2022-09-27T15:25:48.9176400Z ##[endgroup] 2022-09-27T15:25:48.9176901Z ##[group]Setting up auth for fetching submodules 2022-09-27T15:25:48.9182536Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-09-27T15:25:48.9230598Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2022-09-27T15:25:48.9261951Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2022-09-27T15:25:48.9291154Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2022-09-27T15:25:48.9316525Z ##[endgroup] 2022-09-27T15:25:48.9317003Z ##[group]Fetching submodules 2022-09-27T15:25:48.9320422Z [command]/usr/bin/git submodule sync --recursive 2022-09-27T15:25:48.9588050Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2022-09-27T15:25:48.9850072Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2022-09-27T15:25:48.9851318Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2022-09-27T15:25:48.9852469Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2022-09-27T15:25:48.9855421Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2022-09-27T15:25:48.9858400Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2022-09-27T15:25:48.9861859Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2022-09-27T15:25:48.9864785Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2022-09-27T15:25:48.9868091Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2022-09-27T15:25:48.9871640Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2022-09-27T15:25:48.9876127Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2022-09-27T15:25:48.9879331Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2022-09-27T15:25:48.9882957Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2022-09-27T15:25:48.9951910Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2022-09-27T15:25:48.9953167Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2022-09-27T15:25:48.9954286Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2022-09-27T15:25:48.9955434Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2022-09-27T15:25:48.9956443Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2022-09-27T15:25:48.9957400Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2022-09-27T15:25:48.9958355Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2022-09-27T15:25:48.9959483Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2022-09-27T15:25:48.9960418Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2022-09-27T15:25:48.9961351Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2022-09-27T15:25:48.9962417Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2022-09-27T15:25:48.9963377Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2022-09-27T15:25:48.9964251Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2022-09-27T15:25:48.9965191Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2022-09-27T15:25:48.9966267Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2022-09-27T15:25:48.9970616Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2022-09-27T15:25:48.9976121Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2022-09-27T15:25:48.9981646Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2022-09-27T15:25:48.9987361Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2022-09-27T15:25:48.9993263Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2022-09-27T15:25:48.9999375Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2022-09-27T15:25:49.0005435Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2022-09-27T15:25:49.0011450Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum' 2022-09-27T15:25:49.0017674Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2022-09-27T15:25:49.0023942Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six' 2022-09-27T15:25:49.0030210Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2022-09-27T15:25:49.0037026Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2022-09-27T15:25:49.0043845Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2022-09-27T15:25:49.0050471Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2022-09-27T15:25:49.0078069Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2022-09-27T15:25:49.2666435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2022-09-27T15:25:49.4722056Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2022-09-27T15:25:49.6716104Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2022-09-27T15:25:49.9270555Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2022-09-27T15:25:50.1867944Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2022-09-27T15:25:53.3756656Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2022-09-27T15:25:58.8889793Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2022-09-27T15:25:59.4841780Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2022-09-27T15:25:59.9545910Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2022-09-27T15:26:01.3374156Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2022-09-27T15:26:02.5042967Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2022-09-27T15:26:03.7783769Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2022-09-27T15:26:09.7016830Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2022-09-27T15:26:10.3410379Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2022-09-27T15:26:11.6640756Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2022-09-27T15:26:12.7866510Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2022-09-27T15:26:12.9952422Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2022-09-27T15:26:13.4343470Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2022-09-27T15:26:13.7563398Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2022-09-27T15:26:14.8313508Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2022-09-27T15:26:15.2478840Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2022-09-27T15:26:15.5592529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2022-09-27T15:26:15.8098059Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2022-09-27T15:26:17.1487629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2022-09-27T15:26:17.6848018Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2022-09-27T15:26:18.0496240Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2022-09-27T15:26:24.0174588Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2022-09-27T15:26:25.4956451Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2022-09-27T15:26:25.8954882Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2022-09-27T15:26:26.0929477Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2022-09-27T15:26:31.2651365Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2022-09-27T15:26:31.4758627Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2022-09-27T15:26:31.9377146Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2022-09-27T15:26:32.6882196Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'... 2022-09-27T15:26:32.8995743Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2022-09-27T15:26:33.2185449Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'... 2022-09-27T15:26:33.5365532Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2022-09-27T15:26:34.1120035Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2022-09-27T15:26:36.2194321Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2022-09-27T15:26:36.6764689Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2022-09-27T15:26:38.8677544Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2022-09-27T15:26:38.8773409Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2022-09-27T15:26:38.8847786Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2022-09-27T15:26:38.9051312Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2022-09-27T15:26:38.9252003Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2022-09-27T15:26:38.9595603Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2022-09-27T15:26:39.5055510Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba' 2022-09-27T15:26:39.5249412Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-09-27T15:26:39.6158869Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c' 2022-09-27T15:26:39.6476219Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2022-09-27T15:26:39.9117885Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3' 2022-09-27T15:26:40.2845975Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a' 2022-09-27T15:26:40.5079536Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2022-09-27T15:26:40.5508073Z Submodule path 'third_party/fbgemm': checked out '499cd22f5c2e26041e4f190f628b48478a89a030' 2022-09-27T15:26:40.5522385Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2022-09-27T15:26:40.5524501Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2022-09-27T15:26:40.5526984Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2022-09-27T15:26:40.5529714Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2022-09-27T15:26:40.5551929Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2022-09-27T15:26:41.4104838Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2022-09-27T15:26:41.9494289Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2022-09-27T15:26:42.9073883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2022-09-27T15:26:43.2277640Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2022-09-27T15:26:43.3202922Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2022-09-27T15:26:43.3743699Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2022-09-27T15:26:43.3839528Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5' 2022-09-27T15:26:43.4629301Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa' 2022-09-27T15:26:43.4941193Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2022-09-27T15:26:43.5024456Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2022-09-27T15:26:43.5391662Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2022-09-27T15:26:43.5611904Z Submodule path 'third_party/gloo': checked out '5b143513263133af2b95547e97c07cebeb72bf72' 2022-09-27T15:26:43.6031669Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2022-09-27T15:26:43.6134896Z Submodule path 'third_party/ideep': checked out '77d662b313a762e82b389d3fd965e0098f12cd99' 2022-09-27T15:26:43.6147231Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2022-09-27T15:26:43.6167564Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2022-09-27T15:26:50.2137708Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46' 2022-09-27T15:26:50.2153149Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-09-27T15:26:50.2176040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'... 2022-09-27T15:26:58.2078462Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815' 2022-09-27T15:26:58.2172522Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2022-09-27T15:26:58.2300165Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2022-09-27T15:26:58.3159648Z Submodule path 'third_party/kineto': checked out '0703c78999061b8329dfab7ec5046fc5764a5573' 2022-09-27T15:26:58.3174785Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2022-09-27T15:26:58.3176689Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2022-09-27T15:26:58.3199744Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2022-09-27T15:26:59.4553022Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2022-09-27T15:27:00.4378168Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd' 2022-09-27T15:27:00.4874068Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2022-09-27T15:27:00.5054073Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52' 2022-09-27T15:27:00.5183536Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2022-09-27T15:27:00.6164169Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2022-09-27T15:27:00.8400470Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b' 2022-09-27T15:27:00.8426476Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2022-09-27T15:27:00.8428498Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2022-09-27T15:27:00.8453111Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2022-09-27T15:27:01.2839592Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2022-09-27T15:27:02.2061532Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-09-27T15:27:02.2347215Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8' 2022-09-27T15:27:02.2486580Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2022-09-27T15:27:02.2499091Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2022-09-27T15:27:02.2520694Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2022-09-27T15:27:04.1347412Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2022-09-27T15:27:04.1366612Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-09-27T15:27:04.1368522Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-09-27T15:27:04.1390888Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2022-09-27T15:27:04.5580113Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2022-09-27T15:27:05.3425342Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2022-09-27T15:27:05.4011703Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2022-09-27T15:27:05.4024237Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-09-27T15:27:05.4046463Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2022-09-27T15:27:05.6449376Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-09-27T15:27:05.6532988Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2022-09-27T15:27:05.8863820Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2022-09-27T15:27:05.8883599Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2022-09-27T15:27:05.8885484Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2022-09-27T15:27:05.8908057Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2022-09-27T15:27:06.3278164Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2022-09-27T15:27:07.3545345Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2022-09-27T15:27:07.4167694Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2022-09-27T15:27:07.4244976Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2022-09-27T15:27:07.4345003Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2022-09-27T15:27:07.4642836Z Submodule path 'third_party/pybind11': checked out 'aa304c9c7d725ffb9d10af08a3b34cb372307020' 2022-09-27T15:27:07.4723869Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7' 2022-09-27T15:27:07.4986468Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2022-09-27T15:27:07.5074990Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a' 2022-09-27T15:27:07.5481794Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2022-09-27T15:27:07.6473257Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2022-09-27T15:27:07.6717482Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2022-09-27T15:27:07.6732494Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2022-09-27T15:27:07.6734305Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2022-09-27T15:27:07.6736628Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2022-09-27T15:27:07.6739109Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2022-09-27T15:27:07.6762316Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2022-09-27T15:27:08.6835393Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2022-09-27T15:27:08.9481415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2022-09-27T15:27:10.1455153Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2022-09-27T15:27:10.9957065Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2022-09-27T15:27:11.0088481Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2022-09-27T15:27:11.0686460Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2022-09-27T15:27:11.0936798Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2022-09-27T15:27:11.0950863Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-09-27T15:27:11.0972672Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2022-09-27T15:27:11.5217454Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-09-27T15:27:11.6432925Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2022-09-27T15:27:11.6461067Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2022-09-27T15:27:11.6719767Z Entering 'android/libs/fbjni' 2022-09-27T15:27:11.6753480Z Entering 'third_party/FP16' 2022-09-27T15:27:11.6789447Z Entering 'third_party/FXdiv' 2022-09-27T15:27:11.6824165Z Entering 'third_party/NNPACK' 2022-09-27T15:27:11.6860330Z Entering 'third_party/QNNPACK' 2022-09-27T15:27:11.6894891Z Entering 'third_party/VulkanMemoryAllocator' 2022-09-27T15:27:11.6929478Z Entering 'third_party/XNNPACK' 2022-09-27T15:27:11.6974638Z Entering 'third_party/benchmark' 2022-09-27T15:27:11.7008944Z Entering 'third_party/cpuinfo' 2022-09-27T15:27:11.7044040Z Entering 'third_party/cub' 2022-09-27T15:27:11.7077856Z Entering 'third_party/cudnn_frontend' 2022-09-27T15:27:11.7117540Z Entering 'third_party/cutlass' 2022-09-27T15:27:11.7158415Z Entering 'third_party/eigen' 2022-09-27T15:27:11.7195458Z Entering 'third_party/fbgemm' 2022-09-27T15:27:11.7230768Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-09-27T15:27:11.7266029Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-09-27T15:27:11.7299932Z Entering 'third_party/fbgemm/third_party/googletest' 2022-09-27T15:27:11.7333473Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-09-27T15:27:11.7367854Z Entering 'third_party/flatbuffers' 2022-09-27T15:27:11.7405284Z Entering 'third_party/fmt' 2022-09-27T15:27:11.7439509Z Entering 'third_party/foxi' 2022-09-27T15:27:11.7474381Z Entering 'third_party/gemmlowp/gemmlowp' 2022-09-27T15:27:11.7508890Z Entering 'third_party/gloo' 2022-09-27T15:27:11.7542617Z Entering 'third_party/googletest' 2022-09-27T15:27:11.7575889Z Entering 'third_party/ideep' 2022-09-27T15:27:11.7610370Z Entering 'third_party/ideep/mkl-dnn' 2022-09-27T15:27:11.7646428Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-09-27T15:27:11.7686498Z Entering 'third_party/ios-cmake' 2022-09-27T15:27:11.7721156Z Entering 'third_party/ittapi' 2022-09-27T15:27:11.7756880Z Entering 'third_party/kineto' 2022-09-27T15:27:11.7790668Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-09-27T15:27:11.7824719Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-09-27T15:27:11.7860060Z Entering 'third_party/nccl/nccl' 2022-09-27T15:27:11.7897574Z Entering 'third_party/neon2sse' 2022-09-27T15:27:11.7931510Z Entering 'third_party/nlohmann' 2022-09-27T15:27:11.7966751Z Entering 'third_party/onnx' 2022-09-27T15:27:11.8013841Z Entering 'third_party/onnx/third_party/benchmark' 2022-09-27T15:27:11.8047609Z Entering 'third_party/onnx/third_party/pybind11' 2022-09-27T15:27:11.8083472Z Entering 'third_party/onnx-tensorrt' 2022-09-27T15:27:11.8118922Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-09-27T15:27:11.8156781Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-09-27T15:27:11.8190284Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-09-27T15:27:11.8225056Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-09-27T15:27:11.8263522Z Entering 'third_party/pocketfft' 2022-09-27T15:27:11.8297938Z Entering 'third_party/protobuf' 2022-09-27T15:27:11.8336084Z Entering 'third_party/protobuf/third_party/benchmark' 2022-09-27T15:27:11.8370580Z Entering 'third_party/protobuf/third_party/googletest' 2022-09-27T15:27:11.8407063Z Entering 'third_party/psimd' 2022-09-27T15:27:11.8441631Z Entering 'third_party/pthreadpool' 2022-09-27T15:27:11.8476864Z Entering 'third_party/pybind11' 2022-09-27T15:27:11.8511325Z Entering 'third_party/python-enum' 2022-09-27T15:27:11.8545847Z Entering 'third_party/python-peachpy' 2022-09-27T15:27:11.8579885Z Entering 'third_party/python-six' 2022-09-27T15:27:11.8613974Z Entering 'third_party/sleef' 2022-09-27T15:27:11.8650161Z Entering 'third_party/tbb' 2022-09-27T15:27:11.8686507Z Entering 'third_party/tensorpipe' 2022-09-27T15:27:11.8720909Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-09-27T15:27:11.8755641Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-09-27T15:27:11.8789444Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-09-27T15:27:11.8824342Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-09-27T15:27:11.8858218Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-09-27T15:27:11.8895148Z Entering 'third_party/zstd' 2022-09-27T15:27:11.8938661Z ##[endgroup] 2022-09-27T15:27:11.8939148Z ##[group]Persisting credentials for submodules 2022-09-27T15:27:11.8946205Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || : 2022-09-27T15:27:11.9206883Z Entering 'android/libs/fbjni' 2022-09-27T15:27:11.9241256Z Entering 'third_party/FP16' 2022-09-27T15:27:11.9274298Z Entering 'third_party/FXdiv' 2022-09-27T15:27:11.9308284Z Entering 'third_party/NNPACK' 2022-09-27T15:27:11.9341898Z Entering 'third_party/QNNPACK' 2022-09-27T15:27:11.9375883Z Entering 'third_party/VulkanMemoryAllocator' 2022-09-27T15:27:11.9409884Z Entering 'third_party/XNNPACK' 2022-09-27T15:27:11.9453235Z Entering 'third_party/benchmark' 2022-09-27T15:27:11.9486814Z Entering 'third_party/cpuinfo' 2022-09-27T15:27:11.9521557Z Entering 'third_party/cub' 2022-09-27T15:27:11.9554965Z Entering 'third_party/cudnn_frontend' 2022-09-27T15:27:11.9594291Z Entering 'third_party/cutlass' 2022-09-27T15:27:11.9634275Z Entering 'third_party/eigen' 2022-09-27T15:27:11.9670267Z Entering 'third_party/fbgemm' 2022-09-27T15:27:11.9704305Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-09-27T15:27:11.9739239Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-09-27T15:27:11.9773935Z Entering 'third_party/fbgemm/third_party/googletest' 2022-09-27T15:27:11.9809242Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-09-27T15:27:11.9843271Z Entering 'third_party/flatbuffers' 2022-09-27T15:27:11.9880417Z Entering 'third_party/fmt' 2022-09-27T15:27:11.9914276Z Entering 'third_party/foxi' 2022-09-27T15:27:11.9949177Z Entering 'third_party/gemmlowp/gemmlowp' 2022-09-27T15:27:11.9983475Z Entering 'third_party/gloo' 2022-09-27T15:27:12.0018137Z Entering 'third_party/googletest' 2022-09-27T15:27:12.0053178Z Entering 'third_party/ideep' 2022-09-27T15:27:12.0087909Z Entering 'third_party/ideep/mkl-dnn' 2022-09-27T15:27:12.0123943Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-09-27T15:27:12.0164030Z Entering 'third_party/ios-cmake' 2022-09-27T15:27:12.0199322Z Entering 'third_party/ittapi' 2022-09-27T15:27:12.0233638Z Entering 'third_party/kineto' 2022-09-27T15:27:12.0267948Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-09-27T15:27:12.0301948Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-09-27T15:27:12.0338699Z Entering 'third_party/nccl/nccl' 2022-09-27T15:27:12.0373500Z Entering 'third_party/neon2sse' 2022-09-27T15:27:12.0407719Z Entering 'third_party/nlohmann' 2022-09-27T15:27:12.0443211Z Entering 'third_party/onnx' 2022-09-27T15:27:12.0488725Z Entering 'third_party/onnx/third_party/benchmark' 2022-09-27T15:27:12.0523931Z Entering 'third_party/onnx/third_party/pybind11' 2022-09-27T15:27:12.0560826Z Entering 'third_party/onnx-tensorrt' 2022-09-27T15:27:12.0594794Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-09-27T15:27:12.0633265Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-09-27T15:27:12.0667413Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-09-27T15:27:12.0701431Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-09-27T15:27:12.0739623Z Entering 'third_party/pocketfft' 2022-09-27T15:27:12.0774097Z Entering 'third_party/protobuf' 2022-09-27T15:27:12.0811353Z Entering 'third_party/protobuf/third_party/benchmark' 2022-09-27T15:27:12.0845361Z Entering 'third_party/protobuf/third_party/googletest' 2022-09-27T15:27:12.0881886Z Entering 'third_party/psimd' 2022-09-27T15:27:12.0916881Z Entering 'third_party/pthreadpool' 2022-09-27T15:27:12.0951088Z Entering 'third_party/pybind11' 2022-09-27T15:27:12.0984201Z Entering 'third_party/python-enum' 2022-09-27T15:27:12.1019701Z Entering 'third_party/python-peachpy' 2022-09-27T15:27:12.1053291Z Entering 'third_party/python-six' 2022-09-27T15:27:12.1087148Z Entering 'third_party/sleef' 2022-09-27T15:27:12.1120127Z Entering 'third_party/tbb' 2022-09-27T15:27:12.1156350Z Entering 'third_party/tensorpipe' 2022-09-27T15:27:12.1189392Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-09-27T15:27:12.1222258Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-09-27T15:27:12.1255788Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-09-27T15:27:12.1290596Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-09-27T15:27:12.1322885Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-09-27T15:27:12.1359738Z Entering 'third_party/zstd' 2022-09-27T15:27:12.1407082Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url 2022-09-27T15:27:12.1670352Z Entering 'android/libs/fbjni' 2022-09-27T15:27:12.1703502Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2022-09-27T15:27:12.1718563Z Entering 'third_party/FP16' 2022-09-27T15:27:12.1750906Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2022-09-27T15:27:12.1766109Z Entering 'third_party/FXdiv' 2022-09-27T15:27:12.1799120Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2022-09-27T15:27:12.1813270Z Entering 'third_party/NNPACK' 2022-09-27T15:27:12.1848291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2022-09-27T15:27:12.1863587Z Entering 'third_party/QNNPACK' 2022-09-27T15:27:12.1895363Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2022-09-27T15:27:12.1910542Z Entering 'third_party/VulkanMemoryAllocator' 2022-09-27T15:27:12.1942923Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2022-09-27T15:27:12.1958396Z Entering 'third_party/XNNPACK' 2022-09-27T15:27:12.1989452Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2022-09-27T15:27:12.2015171Z Entering 'third_party/benchmark' 2022-09-27T15:27:12.2046334Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2022-09-27T15:27:12.2060762Z Entering 'third_party/cpuinfo' 2022-09-27T15:27:12.2092656Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2022-09-27T15:27:12.2108176Z Entering 'third_party/cub' 2022-09-27T15:27:12.2141046Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2022-09-27T15:27:12.2155064Z Entering 'third_party/cudnn_frontend' 2022-09-27T15:27:12.2186531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2022-09-27T15:27:12.2207209Z Entering 'third_party/cutlass' 2022-09-27T15:27:12.2238611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2022-09-27T15:27:12.2259062Z Entering 'third_party/eigen' 2022-09-27T15:27:12.2290057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2022-09-27T15:27:12.2306724Z Entering 'third_party/fbgemm' 2022-09-27T15:27:12.2338683Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2022-09-27T15:27:12.2353309Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-09-27T15:27:12.2385839Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2022-09-27T15:27:12.2399963Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-09-27T15:27:12.2431770Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2022-09-27T15:27:12.2446606Z Entering 'third_party/fbgemm/third_party/googletest' 2022-09-27T15:27:12.2478447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2022-09-27T15:27:12.2492551Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-09-27T15:27:12.2524917Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2022-09-27T15:27:12.2540659Z Entering 'third_party/flatbuffers' 2022-09-27T15:27:12.2574017Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2022-09-27T15:27:12.2590458Z Entering 'third_party/fmt' 2022-09-27T15:27:12.2623695Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2022-09-27T15:27:12.2638768Z Entering 'third_party/foxi' 2022-09-27T15:27:12.2669863Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2022-09-27T15:27:12.2684752Z Entering 'third_party/gemmlowp/gemmlowp' 2022-09-27T15:27:12.2716148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2022-09-27T15:27:12.2730920Z Entering 'third_party/gloo' 2022-09-27T15:27:12.2771714Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2022-09-27T15:27:12.2777217Z Entering 'third_party/googletest' 2022-09-27T15:27:12.2810376Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2022-09-27T15:27:12.2824329Z Entering 'third_party/ideep' 2022-09-27T15:27:12.2857301Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2022-09-27T15:27:12.2870916Z Entering 'third_party/ideep/mkl-dnn' 2022-09-27T15:27:12.2903240Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2022-09-27T15:27:12.2919709Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-09-27T15:27:12.2951024Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config remote.origin.url 2022-09-27T15:27:12.2973928Z Entering 'third_party/ios-cmake' 2022-09-27T15:27:12.3006399Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2022-09-27T15:27:12.3020368Z Entering 'third_party/ittapi' 2022-09-27T15:27:12.3052457Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2022-09-27T15:27:12.3066397Z Entering 'third_party/kineto' 2022-09-27T15:27:12.3098654Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2022-09-27T15:27:12.3113392Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-09-27T15:27:12.3145509Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2022-09-27T15:27:12.3160529Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-09-27T15:27:12.3191462Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2022-09-27T15:27:12.3208206Z Entering 'third_party/nccl/nccl' 2022-09-27T15:27:12.3240484Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2022-09-27T15:27:12.3254562Z Entering 'third_party/neon2sse' 2022-09-27T15:27:12.3284838Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2022-09-27T15:27:12.3299490Z Entering 'third_party/nlohmann' 2022-09-27T15:27:12.3343375Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2022-09-27T15:27:12.3360003Z Entering 'third_party/onnx' 2022-09-27T15:27:12.3393680Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2022-09-27T15:27:12.3418966Z Entering 'third_party/onnx/third_party/benchmark' 2022-09-27T15:27:12.3451065Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-09-27T15:27:12.3465461Z Entering 'third_party/onnx/third_party/pybind11' 2022-09-27T15:27:12.3497928Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-09-27T15:27:12.3515452Z Entering 'third_party/onnx-tensorrt' 2022-09-27T15:27:12.3546647Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2022-09-27T15:27:12.3562683Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-09-27T15:27:12.3594892Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2022-09-27T15:27:12.3613273Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-09-27T15:27:12.3644413Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-09-27T15:27:12.3659501Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-09-27T15:27:12.3691416Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-09-27T15:27:12.3705777Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-09-27T15:27:12.3737673Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-09-27T15:27:12.3756762Z Entering 'third_party/pocketfft' 2022-09-27T15:27:12.3789010Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2022-09-27T15:27:12.3804233Z Entering 'third_party/protobuf' 2022-09-27T15:27:12.3836612Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2022-09-27T15:27:12.3855873Z Entering 'third_party/protobuf/third_party/benchmark' 2022-09-27T15:27:12.3889753Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2022-09-27T15:27:12.3904372Z Entering 'third_party/protobuf/third_party/googletest' 2022-09-27T15:27:12.3936589Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2022-09-27T15:27:12.3953569Z Entering 'third_party/psimd' 2022-09-27T15:27:12.3986395Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2022-09-27T15:27:12.4001279Z Entering 'third_party/pthreadpool' 2022-09-27T15:27:12.4036012Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2022-09-27T15:27:12.4052372Z Entering 'third_party/pybind11' 2022-09-27T15:27:12.4088520Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2022-09-27T15:27:12.4103943Z Entering 'third_party/python-enum' 2022-09-27T15:27:12.4136713Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config remote.origin.url 2022-09-27T15:27:12.4151584Z Entering 'third_party/python-peachpy' 2022-09-27T15:27:12.4184891Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2022-09-27T15:27:12.4200058Z Entering 'third_party/python-six' 2022-09-27T15:27:12.4233010Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config remote.origin.url 2022-09-27T15:27:12.4246816Z Entering 'third_party/sleef' 2022-09-27T15:27:12.4278798Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2022-09-27T15:27:12.4293446Z Entering 'third_party/tbb' 2022-09-27T15:27:12.4325828Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2022-09-27T15:27:12.4342272Z Entering 'third_party/tensorpipe' 2022-09-27T15:27:12.4374172Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2022-09-27T15:27:12.4388769Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-09-27T15:27:12.4420190Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2022-09-27T15:27:12.4435924Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-09-27T15:27:12.4467638Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2022-09-27T15:27:12.4481704Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-09-27T15:27:12.4513988Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2022-09-27T15:27:12.4528530Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-09-27T15:27:12.4560447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2022-09-27T15:27:12.4574340Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-09-27T15:27:12.4606536Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-09-27T15:27:12.4623879Z Entering 'third_party/zstd' 2022-09-27T15:27:12.4656441Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2022-09-27T15:27:12.6349165Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2022-09-27T15:27:12.6615825Z Entering 'android/libs/fbjni' 2022-09-27T15:27:12.6652534Z Entering 'third_party/FP16' 2022-09-27T15:27:12.6688704Z Entering 'third_party/FXdiv' 2022-09-27T15:27:12.6722349Z Entering 'third_party/NNPACK' 2022-09-27T15:27:12.6760485Z Entering 'third_party/QNNPACK' 2022-09-27T15:27:12.6795436Z Entering 'third_party/VulkanMemoryAllocator' 2022-09-27T15:27:12.6829483Z Entering 'third_party/XNNPACK' 2022-09-27T15:27:12.6873462Z Entering 'third_party/benchmark' 2022-09-27T15:27:12.6907720Z Entering 'third_party/cpuinfo' 2022-09-27T15:27:12.6942379Z Entering 'third_party/cub' 2022-09-27T15:27:12.6976709Z Entering 'third_party/cudnn_frontend' 2022-09-27T15:27:12.7015903Z Entering 'third_party/cutlass' 2022-09-27T15:27:12.7055853Z Entering 'third_party/eigen' 2022-09-27T15:27:12.7092013Z Entering 'third_party/fbgemm' 2022-09-27T15:27:12.7127400Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-09-27T15:27:12.7164755Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-09-27T15:27:12.7200298Z Entering 'third_party/fbgemm/third_party/googletest' 2022-09-27T15:27:12.7235640Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-09-27T15:27:12.7271683Z Entering 'third_party/flatbuffers' 2022-09-27T15:27:12.7310039Z Entering 'third_party/fmt' 2022-09-27T15:27:12.7345158Z Entering 'third_party/foxi' 2022-09-27T15:27:12.7380072Z Entering 'third_party/gemmlowp/gemmlowp' 2022-09-27T15:27:12.7415701Z Entering 'third_party/gloo' 2022-09-27T15:27:12.7450296Z Entering 'third_party/googletest' 2022-09-27T15:27:12.7485862Z Entering 'third_party/ideep' 2022-09-27T15:27:12.7519448Z Entering 'third_party/ideep/mkl-dnn' 2022-09-27T15:27:12.7554738Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-09-27T15:27:12.7596665Z Entering 'third_party/ios-cmake' 2022-09-27T15:27:12.7635313Z Entering 'third_party/ittapi' 2022-09-27T15:27:12.7670112Z Entering 'third_party/kineto' 2022-09-27T15:27:12.7705478Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-09-27T15:27:12.7742989Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-09-27T15:27:12.7778771Z Entering 'third_party/nccl/nccl' 2022-09-27T15:27:12.7815076Z Entering 'third_party/neon2sse' 2022-09-27T15:27:12.7852568Z Entering 'third_party/nlohmann' 2022-09-27T15:27:12.7889590Z Entering 'third_party/onnx' 2022-09-27T15:27:12.7935769Z Entering 'third_party/onnx/third_party/benchmark' 2022-09-27T15:27:12.7969964Z Entering 'third_party/onnx/third_party/pybind11' 2022-09-27T15:27:12.8006501Z Entering 'third_party/onnx-tensorrt' 2022-09-27T15:27:12.8041544Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-09-27T15:27:12.8081167Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-09-27T15:27:12.8116052Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-09-27T15:27:12.8149924Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-09-27T15:27:12.8191392Z Entering 'third_party/pocketfft' 2022-09-27T15:27:12.8227504Z Entering 'third_party/protobuf' 2022-09-27T15:27:12.8265466Z Entering 'third_party/protobuf/third_party/benchmark' 2022-09-27T15:27:12.8300489Z Entering 'third_party/protobuf/third_party/googletest' 2022-09-27T15:27:12.8337625Z Entering 'third_party/psimd' 2022-09-27T15:27:12.8372625Z Entering 'third_party/pthreadpool' 2022-09-27T15:27:12.8407480Z Entering 'third_party/pybind11' 2022-09-27T15:27:12.8443919Z Entering 'third_party/python-enum' 2022-09-27T15:27:12.8480254Z Entering 'third_party/python-peachpy' 2022-09-27T15:27:12.8518676Z Entering 'third_party/python-six' 2022-09-27T15:27:12.8553699Z Entering 'third_party/sleef' 2022-09-27T15:27:12.8589224Z Entering 'third_party/tbb' 2022-09-27T15:27:12.8626485Z Entering 'third_party/tensorpipe' 2022-09-27T15:27:12.8663368Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-09-27T15:27:12.8698677Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-09-27T15:27:12.8732604Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-09-27T15:27:12.8767889Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-09-27T15:27:12.8803653Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-09-27T15:27:12.8841667Z Entering 'third_party/zstd' 2022-09-27T15:27:12.8890649Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2022-09-27T15:27:12.9156413Z Entering 'android/libs/fbjni' 2022-09-27T15:27:12.9188928Z Entering 'third_party/FP16' 2022-09-27T15:27:12.9225458Z Entering 'third_party/FXdiv' 2022-09-27T15:27:12.9260458Z Entering 'third_party/NNPACK' 2022-09-27T15:27:12.9295447Z Entering 'third_party/QNNPACK' 2022-09-27T15:27:12.9332040Z Entering 'third_party/VulkanMemoryAllocator' 2022-09-27T15:27:12.9368112Z Entering 'third_party/XNNPACK' 2022-09-27T15:27:12.9413700Z Entering 'third_party/benchmark' 2022-09-27T15:27:12.9448667Z Entering 'third_party/cpuinfo' 2022-09-27T15:27:12.9485821Z Entering 'third_party/cub' 2022-09-27T15:27:12.9521628Z Entering 'third_party/cudnn_frontend' 2022-09-27T15:27:12.9562770Z Entering 'third_party/cutlass' 2022-09-27T15:27:12.9604999Z Entering 'third_party/eigen' 2022-09-27T15:27:12.9644598Z Entering 'third_party/fbgemm' 2022-09-27T15:27:12.9680024Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-09-27T15:27:12.9715162Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-09-27T15:27:12.9750602Z Entering 'third_party/fbgemm/third_party/googletest' 2022-09-27T15:27:12.9785140Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-09-27T15:27:12.9821922Z Entering 'third_party/flatbuffers' 2022-09-27T15:27:12.9862165Z Entering 'third_party/fmt' 2022-09-27T15:27:12.9897287Z Entering 'third_party/foxi' 2022-09-27T15:27:12.9935377Z Entering 'third_party/gemmlowp/gemmlowp' 2022-09-27T15:27:12.9970577Z Entering 'third_party/gloo' 2022-09-27T15:27:13.0008393Z Entering 'third_party/googletest' 2022-09-27T15:27:13.0043828Z Entering 'third_party/ideep' 2022-09-27T15:27:13.0079134Z Entering 'third_party/ideep/mkl-dnn' 2022-09-27T15:27:13.0115990Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-09-27T15:27:13.0157086Z Entering 'third_party/ios-cmake' 2022-09-27T15:27:13.0192728Z Entering 'third_party/ittapi' 2022-09-27T15:27:13.0227675Z Entering 'third_party/kineto' 2022-09-27T15:27:13.0262264Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-09-27T15:27:13.0297348Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-09-27T15:27:13.0332150Z Entering 'third_party/nccl/nccl' 2022-09-27T15:27:13.0367271Z Entering 'third_party/neon2sse' 2022-09-27T15:27:13.0401791Z Entering 'third_party/nlohmann' 2022-09-27T15:27:13.0436944Z Entering 'third_party/onnx' 2022-09-27T15:27:13.0483816Z Entering 'third_party/onnx/third_party/benchmark' 2022-09-27T15:27:13.0517547Z Entering 'third_party/onnx/third_party/pybind11' 2022-09-27T15:27:13.0555062Z Entering 'third_party/onnx-tensorrt' 2022-09-27T15:27:13.0589142Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-09-27T15:27:13.0627400Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-09-27T15:27:13.0662363Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-09-27T15:27:13.0696565Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-09-27T15:27:13.0735170Z Entering 'third_party/pocketfft' 2022-09-27T15:27:13.0769227Z Entering 'third_party/protobuf' 2022-09-27T15:27:13.0806717Z Entering 'third_party/protobuf/third_party/benchmark' 2022-09-27T15:27:13.0839427Z Entering 'third_party/protobuf/third_party/googletest' 2022-09-27T15:27:13.0875026Z Entering 'third_party/psimd' 2022-09-27T15:27:13.0909421Z Entering 'third_party/pthreadpool' 2022-09-27T15:27:13.0944146Z Entering 'third_party/pybind11' 2022-09-27T15:27:13.0978989Z Entering 'third_party/python-enum' 2022-09-27T15:27:13.1013303Z Entering 'third_party/python-peachpy' 2022-09-27T15:27:13.1047816Z Entering 'third_party/python-six' 2022-09-27T15:27:13.1084396Z Entering 'third_party/sleef' 2022-09-27T15:27:13.1119592Z Entering 'third_party/tbb' 2022-09-27T15:27:13.1156662Z Entering 'third_party/tensorpipe' 2022-09-27T15:27:13.1192845Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-09-27T15:27:13.1226201Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-09-27T15:27:13.1261660Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-09-27T15:27:13.1295984Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-09-27T15:27:13.1332374Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-09-27T15:27:13.1371429Z Entering 'third_party/zstd' 2022-09-27T15:27:13.1414082Z ##[endgroup] 2022-09-27T15:27:13.1454789Z [command]/usr/bin/git log -1 --format='%H' 2022-09-27T15:27:13.1480973Z '52424e2bf38e454d535881fed9628d3e20f4f944' 2022-09-27T15:27:13.1720098Z Prepare all required actions 2022-09-27T15:27:13.1745880Z ##[group]Run ./.github/actions/setup-linux 2022-09-27T15:27:13.1746079Z env: 2022-09-27T15:27:13.1746241Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:27:13.1746427Z ##[endgroup] 2022-09-27T15:27:13.1845915Z ##[group]Run set -euo pipefail 2022-09-27T15:27:13.1846151Z set -euo pipefail 2022-09-27T15:27:13.1846351Z function get_ec2_metadata() { 2022-09-27T15:27:13.1846599Z  # Pulled from instance metadata endpoint for EC2 2022-09-27T15:27:13.1846955Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2022-09-27T15:27:13.1847240Z  category=$1 2022-09-27T15:27:13.1847484Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2022-09-27T15:27:13.1847711Z } 2022-09-27T15:27:13.1847913Z echo "ami-id: $(get_ec2_metadata ami-id)" 2022-09-27T15:27:13.1848191Z echo "instance-id: $(get_ec2_metadata instance-id)" 2022-09-27T15:27:13.1848560Z echo "instance-type: $(get_ec2_metadata instance-type)" 2022-09-27T15:27:13.1848812Z echo "system info $(uname -a)" 2022-09-27T15:27:13.1860802Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T15:27:13.1861005Z env: 2022-09-27T15:27:13.1861180Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:27:13.1861370Z ##[endgroup] 2022-09-27T15:27:13.1939871Z ami-id: ami-096198a0bccc6bad4 2022-09-27T15:27:13.1988559Z instance-id: i-00a9b5b3060f0b9ae 2022-09-27T15:27:13.2036777Z instance-type: c5.2xlarge 2022-09-27T15:27:13.2044341Z system info Linux ip-10-0-3-234.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 2022-09-27T15:27:13.2060037Z ##[group]Run if systemctl is-active --quiet docker; then 2022-09-27T15:27:13.2060314Z if systemctl is-active --quiet docker; then 2022-09-27T15:27:13.2060566Z  echo "Docker daemon is running..."; 2022-09-27T15:27:13.2060781Z else 2022-09-27T15:27:13.2061018Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2022-09-27T15:27:13.2061234Z fi 2022-09-27T15:27:13.2071704Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T15:27:13.2071922Z env: 2022-09-27T15:27:13.2072088Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:27:13.2072281Z ##[endgroup] 2022-09-27T15:27:13.2112500Z Docker daemon is running... 2022-09-27T15:27:13.2129862Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") 2022-09-27T15:27:13.2130222Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") 2022-09-27T15:27:13.2130507Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-09-27T15:27:13.2130893Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \ 2022-09-27T15:27:13.2131230Z  --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2022-09-27T15:27:13.2142363Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T15:27:13.2142580Z env: 2022-09-27T15:27:13.2142745Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:27:13.2142944Z AWS_RETRY_MODE: standard 2022-09-27T15:27:13.2143131Z AWS_MAX_ATTEMPTS: 5 2022-09-27T15:27:13.2143315Z AWS_DEFAULT_REGION: us-east-1 2022-09-27T15:27:13.2143503Z ##[endgroup] 2022-09-27T15:27:13.9749439Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2022-09-27T15:27:13.9749812Z Configure a credential helper to remove this warning. See 2022-09-27T15:27:13.9750211Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2022-09-27T15:27:13.9750458Z 2022-09-27T15:27:13.9750951Z Login Succeeded 2022-09-27T15:27:13.9783650Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-09-27T15:27:13.9783960Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-09-27T15:27:13.9784302Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-09-27T15:27:13.9795487Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T15:27:13.9795713Z env: 2022-09-27T15:27:13.9795891Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:27:13.9796074Z ##[endgroup] 2022-09-27T15:27:13.9857136Z Prepare all required actions 2022-09-27T15:27:13.9857511Z Getting action download info 2022-09-27T15:27:14.1545212Z Download action repository 'seemethere/add-github-ssh-key@v1' (SHA:105f7619adc4054f5f1be5f79ebd354d82384638) 2022-09-27T15:27:14.2763354Z ##[group]Run ./.github/actions/setup-ssh 2022-09-27T15:27:14.2763558Z with: 2022-09-27T15:27:14.2763889Z github-secret: *** 2022-09-27T15:27:14.2764046Z env: 2022-09-27T15:27:14.2764221Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:27:14.2764406Z ##[endgroup] 2022-09-27T15:27:14.2805539Z ##[group]Run seemethere/add-github-ssh-key@v1 2022-09-27T15:27:14.2805752Z with: 2022-09-27T15:27:14.2806041Z GITHUB_TOKEN: *** 2022-09-27T15:27:14.2806221Z activate-with-label: false 2022-09-27T15:27:14.2806482Z label: with-ssh 2022-09-27T15:27:14.2806671Z remove-existing-keys: true 2022-09-27T15:27:14.2806839Z env: 2022-09-27T15:27:14.2807008Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:27:14.2807193Z ##[endgroup] 2022-09-27T15:27:14.6578060Z Grabbing public ssh keys from https://github.com/kongzii.keys 2022-09-27T15:27:14.7400735Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2022-09-27T15:27:14.7415190Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2022-09-27T15:27:14.7437208Z Login using: ssh ec2-user@ec2-54-221-56-29.compute-1.amazonaws.com 2022-09-27T15:27:14.7495527Z Prepare all required actions 2022-09-27T15:27:14.7557549Z ##[group]Run ./.github/actions/pull-docker-image 2022-09-27T15:27:14.7557852Z with: 2022-09-27T15:27:14.7558216Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29 2022-09-27T15:27:14.7558532Z env: 2022-09-27T15:27:14.7558712Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:27:14.7558900Z ##[endgroup] 2022-09-27T15:27:14.7593887Z ##[group]Run retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-09-27T15:27:14.7594162Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-09-27T15:27:14.7594458Z # ignore output since only exit code is used for conditional 2022-09-27T15:27:14.7594883Z # only pull docker image if it's not available locally 2022-09-27T15:27:14.7595178Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2022-09-27T15:27:14.7595479Z  retry docker pull "${DOCKER_IMAGE}" 2022-09-27T15:27:14.7595660Z fi 2022-09-27T15:27:14.7606944Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T15:27:14.7607160Z env: 2022-09-27T15:27:14.7607335Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:27:14.7607681Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29 2022-09-27T15:27:14.7608019Z ##[endgroup] 2022-09-27T15:27:14.9857166Z e66cf5fa0a4d4ed512901b12ccdab95cca946a29: Pulling from pytorch/pytorch-linux-focal-py3.7-gcc7 2022-09-27T15:27:14.9869339Z 675920708c8b: Pulling fs layer 2022-09-27T15:27:14.9869894Z 7249ff1ca2cc: Pulling fs layer 2022-09-27T15:27:14.9870453Z a1a4ad211d43: Pulling fs layer 2022-09-27T15:27:14.9871107Z dac0b0909394: Pulling fs layer 2022-09-27T15:27:14.9871462Z 4e5e6fbf81d0: Pulling fs layer 2022-09-27T15:27:14.9871884Z 1fd57c418b4f: Pulling fs layer 2022-09-27T15:27:14.9872223Z c252dd0cf292: Pulling fs layer 2022-09-27T15:27:14.9872558Z 6d1cf6420de5: Pulling fs layer 2022-09-27T15:27:14.9872899Z 22ff1eeb7cfc: Pulling fs layer 2022-09-27T15:27:14.9873205Z d8d2312a3e95: Pulling fs layer 2022-09-27T15:27:14.9873537Z 6b8fe070c229: Pulling fs layer 2022-09-27T15:27:14.9873863Z f860349178a3: Pulling fs layer 2022-09-27T15:27:14.9874184Z b9600c108e0a: Pulling fs layer 2022-09-27T15:27:14.9874682Z d7379e9f0242: Pulling fs layer 2022-09-27T15:27:14.9875012Z 111b3362d2ab: Pulling fs layer 2022-09-27T15:27:14.9875300Z 1fd57c418b4f: Waiting 2022-09-27T15:27:14.9875583Z 357e25b632e3: Pulling fs layer 2022-09-27T15:27:14.9875901Z 041ccbc6424a: Pulling fs layer 2022-09-27T15:27:14.9876203Z c252dd0cf292: Waiting 2022-09-27T15:27:14.9876491Z f83a7aa0ae33: Pulling fs layer 2022-09-27T15:27:14.9876863Z 22ff1eeb7cfc: Waiting 2022-09-27T15:27:14.9877154Z 6d1cf6420de5: Waiting 2022-09-27T15:27:14.9877440Z 7dd92e4c6366: Pulling fs layer 2022-09-27T15:27:14.9877766Z 4c3fc199e49b: Pulling fs layer 2022-09-27T15:27:14.9878084Z 1cf2d121678f: Pulling fs layer 2022-09-27T15:27:14.9878361Z 3d0511d3c79c: Pulling fs layer 2022-09-27T15:27:14.9878694Z 14b2bb0370e4: Pulling fs layer 2022-09-27T15:27:14.9879018Z 8da4048e870a: Pulling fs layer 2022-09-27T15:27:14.9879307Z 6b8fe070c229: Waiting 2022-09-27T15:27:14.9879585Z df7557c6da37: Pulling fs layer 2022-09-27T15:27:14.9879881Z f860349178a3: Waiting 2022-09-27T15:27:14.9880336Z 357e25b632e3: Waiting 2022-09-27T15:27:14.9880605Z c47c4a7e8472: Pulling fs layer 2022-09-27T15:27:14.9880885Z d7379e9f0242: Waiting 2022-09-27T15:27:14.9881170Z 041ccbc6424a: Waiting 2022-09-27T15:27:14.9881470Z fbbb9ed3e845: Pulling fs layer 2022-09-27T15:27:14.9881771Z 111b3362d2ab: Waiting 2022-09-27T15:27:14.9882053Z f83a7aa0ae33: Waiting 2022-09-27T15:27:14.9882314Z b9600c108e0a: Waiting 2022-09-27T15:27:14.9882627Z 25c87328e320: Pulling fs layer 2022-09-27T15:27:14.9882937Z 7dd92e4c6366: Waiting 2022-09-27T15:27:14.9883239Z 6ce6383c8b4c: Pulling fs layer 2022-09-27T15:27:14.9883579Z 805e81bf1b07: Pulling fs layer 2022-09-27T15:27:14.9884107Z dac0b0909394: Waiting 2022-09-27T15:27:14.9884415Z 658a981e4c72: Pulling fs layer 2022-09-27T15:27:14.9884733Z 4c3fc199e49b: Waiting 2022-09-27T15:27:14.9885005Z 8da4048e870a: Waiting 2022-09-27T15:27:14.9885295Z 6e614c1c92e2: Pulling fs layer 2022-09-27T15:27:14.9885592Z 1cf2d121678f: Waiting 2022-09-27T15:27:14.9885901Z ca8c741a00af: Pulling fs layer 2022-09-27T15:27:14.9886241Z ec20b9c0ab8c: Pulling fs layer 2022-09-27T15:27:14.9886582Z 4778e25ffa15: Pulling fs layer 2022-09-27T15:27:14.9886910Z 6178951b0ba1: Pulling fs layer 2022-09-27T15:27:14.9887194Z df7557c6da37: Waiting 2022-09-27T15:27:14.9887493Z 5f80c355afff: Pulling fs layer 2022-09-27T15:27:14.9887808Z c47c4a7e8472: Waiting 2022-09-27T15:27:14.9888089Z 8fd17095407b: Pulling fs layer 2022-09-27T15:27:14.9888405Z 7d3e30f68342: Pulling fs layer 2022-09-27T15:27:14.9888730Z fbbb9ed3e845: Waiting 2022-09-27T15:27:14.9889032Z 4e5e6fbf81d0: Waiting 2022-09-27T15:27:14.9889356Z df7d91b50502: Pulling fs layer 2022-09-27T15:27:14.9889697Z ddc86fc3214e: Pulling fs layer 2022-09-27T15:27:14.9890020Z 46ee14c7b526: Pulling fs layer 2022-09-27T15:27:14.9890352Z 9ce9a24e805b: Pulling fs layer 2022-09-27T15:27:14.9890659Z 25c87328e320: Waiting 2022-09-27T15:27:14.9890936Z 19cd914bfdcb: Pulling fs layer 2022-09-27T15:27:14.9891255Z 6e614c1c92e2: Waiting 2022-09-27T15:27:14.9891574Z a7f4338ecba2: Pulling fs layer 2022-09-27T15:27:14.9891876Z 5f80c355afff: Waiting 2022-09-27T15:27:14.9892181Z ca8c741a00af: Waiting 2022-09-27T15:27:14.9892507Z c75978fac091: Pulling fs layer 2022-09-27T15:27:14.9892846Z f89a91cff893: Pulling fs layer 2022-09-27T15:27:14.9893171Z f345365236d1: Pulling fs layer 2022-09-27T15:27:14.9893484Z 6ce6383c8b4c: Waiting 2022-09-27T15:27:14.9893801Z 4a3f3d395eb7: Pulling fs layer 2022-09-27T15:27:14.9894106Z ec20b9c0ab8c: Waiting 2022-09-27T15:27:14.9894430Z 805e81bf1b07: Waiting 2022-09-27T15:27:14.9894726Z 4778e25ffa15: Waiting 2022-09-27T15:27:14.9895006Z e421c81bb4cf: Pulling fs layer 2022-09-27T15:27:14.9895312Z 775c0184544e: Pulling fs layer 2022-09-27T15:27:14.9895623Z 658a981e4c72: Waiting 2022-09-27T15:27:14.9895901Z 5a69da2170e4: Pulling fs layer 2022-09-27T15:27:14.9896199Z df7d91b50502: Waiting 2022-09-27T15:27:14.9896494Z d3273d80d5f9: Pulling fs layer 2022-09-27T15:27:14.9896798Z 6178951b0ba1: Waiting 2022-09-27T15:27:14.9897106Z ddc86fc3214e: Waiting 2022-09-27T15:27:14.9897438Z f5905b45d58c: Pulling fs layer 2022-09-27T15:27:14.9897767Z 42b9d02069e0: Pulling fs layer 2022-09-27T15:27:14.9898079Z 46ee14c7b526: Waiting 2022-09-27T15:27:14.9898380Z 9ce9a24e805b: Waiting 2022-09-27T15:27:14.9898683Z c00b10f24e83: Pulling fs layer 2022-09-27T15:27:14.9899009Z 19cd914bfdcb: Waiting 2022-09-27T15:27:14.9899334Z 93c6c1ce6a86: Pulling fs layer 2022-09-27T15:27:14.9899656Z 239233a60227: Pulling fs layer 2022-09-27T15:27:14.9899962Z a7f4338ecba2: Waiting 2022-09-27T15:27:14.9900270Z 4a3f3d395eb7: Waiting 2022-09-27T15:27:14.9900558Z 14b2bb0370e4: Waiting 2022-09-27T15:27:14.9900857Z c75978fac091: Waiting 2022-09-27T15:27:14.9901160Z f89a91cff893: Waiting 2022-09-27T15:27:14.9901431Z 5a69da2170e4: Waiting 2022-09-27T15:27:14.9901732Z f345365236d1: Waiting 2022-09-27T15:27:14.9902000Z c00b10f24e83: Waiting 2022-09-27T15:27:14.9902277Z 93c6c1ce6a86: Waiting 2022-09-27T15:27:14.9902560Z 775c0184544e: Waiting 2022-09-27T15:27:14.9902843Z 239233a60227: Waiting 2022-09-27T15:27:14.9903126Z 7d3e30f68342: Waiting 2022-09-27T15:27:14.9903513Z 42b9d02069e0: Waiting 2022-09-27T15:27:14.9903777Z 3d0511d3c79c: Waiting 2022-09-27T15:27:15.0642492Z 7249ff1ca2cc: Verifying Checksum 2022-09-27T15:27:15.0642974Z 7249ff1ca2cc: Download complete 2022-09-27T15:27:15.1761480Z dac0b0909394: Verifying Checksum 2022-09-27T15:27:15.1761809Z dac0b0909394: Download complete 2022-09-27T15:27:15.2585897Z 4e5e6fbf81d0: Verifying Checksum 2022-09-27T15:27:15.2586342Z 4e5e6fbf81d0: Download complete 2022-09-27T15:27:15.3276672Z 1fd57c418b4f: Download complete 2022-09-27T15:27:15.3282558Z 675920708c8b: Verifying Checksum 2022-09-27T15:27:15.3283086Z 675920708c8b: Download complete 2022-09-27T15:27:15.4035461Z c252dd0cf292: Verifying Checksum 2022-09-27T15:27:15.4036000Z c252dd0cf292: Download complete 2022-09-27T15:27:15.4165684Z 6d1cf6420de5: Verifying Checksum 2022-09-27T15:27:15.4166077Z 6d1cf6420de5: Download complete 2022-09-27T15:27:15.4988257Z d8d2312a3e95: Verifying Checksum 2022-09-27T15:27:15.4988640Z d8d2312a3e95: Download complete 2022-09-27T15:27:15.5060238Z 22ff1eeb7cfc: Verifying Checksum 2022-09-27T15:27:15.5062620Z 22ff1eeb7cfc: Download complete 2022-09-27T15:27:15.5839060Z f860349178a3: Verifying Checksum 2022-09-27T15:27:15.5839459Z f860349178a3: Download complete 2022-09-27T15:27:15.6683419Z b9600c108e0a: Verifying Checksum 2022-09-27T15:27:15.6684295Z b9600c108e0a: Download complete 2022-09-27T15:27:16.0203032Z 675920708c8b: Pull complete 2022-09-27T15:27:16.1240614Z 7249ff1ca2cc: Pull complete 2022-09-27T15:27:16.3782485Z 6b8fe070c229: Verifying Checksum 2022-09-27T15:27:16.3782868Z 6b8fe070c229: Download complete 2022-09-27T15:27:16.4546336Z 111b3362d2ab: Verifying Checksum 2022-09-27T15:27:16.4546631Z 111b3362d2ab: Download complete 2022-09-27T15:27:16.5351191Z 357e25b632e3: Verifying Checksum 2022-09-27T15:27:16.5351659Z 357e25b632e3: Download complete 2022-09-27T15:27:17.0306967Z 041ccbc6424a: Verifying Checksum 2022-09-27T15:27:17.0307277Z 041ccbc6424a: Download complete 2022-09-27T15:27:17.1118200Z f83a7aa0ae33: Verifying Checksum 2022-09-27T15:27:17.1118486Z f83a7aa0ae33: Download complete 2022-09-27T15:27:17.2221734Z 7dd92e4c6366: Download complete 2022-09-27T15:27:17.3143649Z 4c3fc199e49b: Download complete 2022-09-27T15:27:17.3896935Z 1cf2d121678f: Download complete 2022-09-27T15:27:17.4670489Z 3d0511d3c79c: Verifying Checksum 2022-09-27T15:27:17.4670764Z 3d0511d3c79c: Download complete 2022-09-27T15:27:17.5467442Z 14b2bb0370e4: Verifying Checksum 2022-09-27T15:27:17.5467861Z 14b2bb0370e4: Download complete 2022-09-27T15:27:17.6275683Z 8da4048e870a: Verifying Checksum 2022-09-27T15:27:17.6276000Z 8da4048e870a: Download complete 2022-09-27T15:27:18.0993044Z a1a4ad211d43: Download complete 2022-09-27T15:27:18.1828383Z c47c4a7e8472: Verifying Checksum 2022-09-27T15:27:18.1828758Z c47c4a7e8472: Download complete 2022-09-27T15:27:18.2635996Z fbbb9ed3e845: Verifying Checksum 2022-09-27T15:27:18.2637854Z fbbb9ed3e845: Download complete 2022-09-27T15:27:18.3571238Z 25c87328e320: Verifying Checksum 2022-09-27T15:27:18.3571652Z 25c87328e320: Download complete 2022-09-27T15:27:18.4837312Z 6ce6383c8b4c: Verifying Checksum 2022-09-27T15:27:18.4837687Z 6ce6383c8b4c: Download complete 2022-09-27T15:27:18.5669352Z 805e81bf1b07: Download complete 2022-09-27T15:27:18.5794747Z df7557c6da37: Verifying Checksum 2022-09-27T15:27:18.5795121Z df7557c6da37: Download complete 2022-09-27T15:27:18.6520046Z 6e614c1c92e2: Download complete 2022-09-27T15:27:18.7415994Z ca8c741a00af: Verifying Checksum 2022-09-27T15:27:18.7416436Z ca8c741a00af: Download complete 2022-09-27T15:27:18.8132211Z ec20b9c0ab8c: Verifying Checksum 2022-09-27T15:27:18.8132496Z ec20b9c0ab8c: Download complete 2022-09-27T15:27:18.9097102Z 4778e25ffa15: Verifying Checksum 2022-09-27T15:27:18.9097438Z 4778e25ffa15: Download complete 2022-09-27T15:27:18.9853133Z 6178951b0ba1: Verifying Checksum 2022-09-27T15:27:18.9853405Z 6178951b0ba1: Download complete 2022-09-27T15:27:19.0544078Z 5f80c355afff: Verifying Checksum 2022-09-27T15:27:19.1302221Z 5f80c355afff: Download complete 2022-09-27T15:27:19.1302794Z 8fd17095407b: Verifying Checksum 2022-09-27T15:27:19.1303132Z 8fd17095407b: Download complete 2022-09-27T15:27:19.2055320Z 7d3e30f68342: Verifying Checksum 2022-09-27T15:27:19.2055699Z 7d3e30f68342: Download complete 2022-09-27T15:27:19.2844930Z df7d91b50502: Verifying Checksum 2022-09-27T15:27:19.2845319Z df7d91b50502: Download complete 2022-09-27T15:27:19.3605629Z ddc86fc3214e: Download complete 2022-09-27T15:27:19.4453393Z 46ee14c7b526: Verifying Checksum 2022-09-27T15:27:19.4453796Z 46ee14c7b526: Download complete 2022-09-27T15:27:19.5107673Z 9ce9a24e805b: Download complete 2022-09-27T15:27:19.9743592Z 19cd914bfdcb: Verifying Checksum 2022-09-27T15:27:19.9743978Z 19cd914bfdcb: Download complete 2022-09-27T15:27:20.0633006Z a7f4338ecba2: Verifying Checksum 2022-09-27T15:27:20.0633279Z a7f4338ecba2: Download complete 2022-09-27T15:27:20.1437273Z c75978fac091: Verifying Checksum 2022-09-27T15:27:20.1437611Z c75978fac091: Download complete 2022-09-27T15:27:20.2213998Z f89a91cff893: Verifying Checksum 2022-09-27T15:27:20.2214997Z f89a91cff893: Download complete 2022-09-27T15:27:20.3437921Z f345365236d1: Verifying Checksum 2022-09-27T15:27:20.3438194Z f345365236d1: Download complete 2022-09-27T15:27:20.6180541Z 4a3f3d395eb7: Verifying Checksum 2022-09-27T15:27:20.6180914Z 4a3f3d395eb7: Download complete 2022-09-27T15:27:20.7053135Z e421c81bb4cf: Verifying Checksum 2022-09-27T15:27:20.7053581Z e421c81bb4cf: Download complete 2022-09-27T15:27:20.7887096Z 775c0184544e: Download complete 2022-09-27T15:27:20.8126185Z 658a981e4c72: Verifying Checksum 2022-09-27T15:27:20.8126494Z 658a981e4c72: Download complete 2022-09-27T15:27:20.8958542Z d3273d80d5f9: Download complete 2022-09-27T15:27:21.0250632Z f5905b45d58c: Download complete 2022-09-27T15:27:21.1089929Z 42b9d02069e0: Download complete 2022-09-27T15:27:21.2140345Z c00b10f24e83: Verifying Checksum 2022-09-27T15:27:21.2140746Z c00b10f24e83: Download complete 2022-09-27T15:27:21.2682727Z 93c6c1ce6a86: Download complete 2022-09-27T15:27:21.8745169Z 239233a60227: Verifying Checksum 2022-09-27T15:27:21.8745560Z 239233a60227: Download complete 2022-09-27T15:27:23.5480939Z 5a69da2170e4: Verifying Checksum 2022-09-27T15:27:23.5481767Z 5a69da2170e4: Download complete 2022-09-27T15:27:25.1705274Z a1a4ad211d43: Pull complete 2022-09-27T15:27:25.3830084Z dac0b0909394: Pull complete 2022-09-27T15:27:25.6901180Z 4e5e6fbf81d0: Pull complete 2022-09-27T15:27:25.9193204Z 1fd57c418b4f: Pull complete 2022-09-27T15:27:26.1258581Z c252dd0cf292: Pull complete 2022-09-27T15:27:26.3278195Z 6d1cf6420de5: Pull complete 2022-09-27T15:27:26.5425842Z 22ff1eeb7cfc: Pull complete 2022-09-27T15:27:26.7746318Z d8d2312a3e95: Pull complete 2022-09-27T15:27:28.5689393Z 6b8fe070c229: Pull complete 2022-09-27T15:27:28.7531291Z f860349178a3: Pull complete 2022-09-27T15:27:29.0013914Z b9600c108e0a: Pull complete 2022-09-27T15:27:33.5055721Z d7379e9f0242: Verifying Checksum 2022-09-27T15:27:33.5056122Z d7379e9f0242: Download complete 2022-09-27T15:27:57.7633577Z d7379e9f0242: Pull complete 2022-09-27T15:27:58.0739560Z 111b3362d2ab: Pull complete 2022-09-27T15:27:58.1659820Z 357e25b632e3: Pull complete 2022-09-27T15:27:58.9288267Z 041ccbc6424a: Pull complete 2022-09-27T15:27:59.0275900Z f83a7aa0ae33: Pull complete 2022-09-27T15:27:59.1583981Z 7dd92e4c6366: Pull complete 2022-09-27T15:27:59.2664810Z 4c3fc199e49b: Pull complete 2022-09-27T15:27:59.3586917Z 1cf2d121678f: Pull complete 2022-09-27T15:27:59.4419033Z 3d0511d3c79c: Pull complete 2022-09-27T15:27:59.5268452Z 14b2bb0370e4: Pull complete 2022-09-27T15:27:59.6152315Z 8da4048e870a: Pull complete 2022-09-27T15:28:01.4978201Z df7557c6da37: Pull complete 2022-09-27T15:28:01.6537006Z c47c4a7e8472: Pull complete 2022-09-27T15:28:01.8644366Z fbbb9ed3e845: Pull complete 2022-09-27T15:28:02.1420812Z 25c87328e320: Pull complete 2022-09-27T15:28:02.3812796Z 6ce6383c8b4c: Pull complete 2022-09-27T15:28:02.6167601Z 805e81bf1b07: Pull complete 2022-09-27T15:28:07.1197654Z 658a981e4c72: Pull complete 2022-09-27T15:28:07.3694402Z 6e614c1c92e2: Pull complete 2022-09-27T15:28:07.6068600Z ca8c741a00af: Pull complete 2022-09-27T15:28:07.8294524Z ec20b9c0ab8c: Pull complete 2022-09-27T15:28:08.0340506Z 4778e25ffa15: Pull complete 2022-09-27T15:28:08.2631579Z 6178951b0ba1: Pull complete 2022-09-27T15:28:08.4855776Z 5f80c355afff: Pull complete 2022-09-27T15:28:08.5738046Z 8fd17095407b: Pull complete 2022-09-27T15:28:08.6836379Z 7d3e30f68342: Pull complete 2022-09-27T15:28:08.7781101Z df7d91b50502: Pull complete 2022-09-27T15:28:08.8753769Z ddc86fc3214e: Pull complete 2022-09-27T15:28:08.9544897Z 46ee14c7b526: Pull complete 2022-09-27T15:28:09.0377782Z 9ce9a24e805b: Pull complete 2022-09-27T15:28:10.1323258Z 19cd914bfdcb: Pull complete 2022-09-27T15:28:10.2317681Z a7f4338ecba2: Pull complete 2022-09-27T15:28:10.3302309Z c75978fac091: Pull complete 2022-09-27T15:28:10.4301455Z f89a91cff893: Pull complete 2022-09-27T15:28:10.5170037Z f345365236d1: Pull complete 2022-09-27T15:28:10.8140806Z 4a3f3d395eb7: Pull complete 2022-09-27T15:28:10.9085231Z e421c81bb4cf: Pull complete 2022-09-27T15:28:11.0170951Z 775c0184544e: Pull complete 2022-09-27T15:28:14.7348890Z 5a69da2170e4: Pull complete 2022-09-27T15:28:14.9556025Z d3273d80d5f9: Pull complete 2022-09-27T15:28:15.0423943Z f5905b45d58c: Pull complete 2022-09-27T15:28:15.1268230Z 42b9d02069e0: Pull complete 2022-09-27T15:28:15.2141746Z c00b10f24e83: Pull complete 2022-09-27T15:28:15.2902135Z 93c6c1ce6a86: Pull complete 2022-09-27T15:28:16.8151016Z 239233a60227: Pull complete 2022-09-27T15:28:16.8299129Z Digest: sha256:aa31a48de2d62af84cacd725abaa1599e63c6bb3385fa6e176b6d91f90a39348 2022-09-27T15:28:16.8354209Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29 2022-09-27T15:28:16.8380013Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29 2022-09-27T15:28:16.8434875Z ##[group]Run python3 -m pip install psutil==5.9.1 2022-09-27T15:28:16.8435176Z python3 -m pip install psutil==5.9.1 2022-09-27T15:28:16.8435420Z python3 -m pip install pynvml==11.4.1 2022-09-27T15:28:16.8435683Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2022-09-27T15:28:16.8435953Z echo "::set-output name=monitor-script-pid::${!}" 2022-09-27T15:28:16.8447126Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T15:28:16.8447342Z env: 2022-09-27T15:28:16.8447514Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:28:16.8447689Z ##[endgroup] 2022-09-27T15:28:17.4072131Z Defaulting to user installation because normal site-packages is not writeable 2022-09-27T15:28:17.4289721Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2022-09-27T15:28:17.8359633Z Defaulting to user installation because normal site-packages is not writeable 2022-09-27T15:28:17.8536960Z Requirement already satisfied: pynvml==11.4.1 in /home/ec2-user/.local/lib/python3.7/site-packages (11.4.1) 2022-09-27T15:28:18.0413157Z Prepare all required actions 2022-09-27T15:28:18.0413412Z Getting action download info 2022-09-27T15:28:18.2996140Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:ada9688bc02703b63dc0e606da280613803449a5) 2022-09-27T15:28:18.5284145Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869) 2022-09-27T15:28:19.6070853Z ##[group]Run ./.github/actions/download-build-artifacts 2022-09-27T15:28:19.6071086Z with: 2022-09-27T15:28:19.6071266Z name: linux-focal-py3.7-gcc7 2022-09-27T15:28:19.6071557Z env: 2022-09-27T15:28:19.6071735Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:28:19.6071910Z ##[endgroup] 2022-09-27T15:28:19.6203977Z ##[group]Run seemethere/download-artifact-s3@v4 2022-09-27T15:28:19.6204194Z with: 2022-09-27T15:28:19.6204381Z name: linux-focal-py3.7-gcc7 2022-09-27T15:28:19.6204575Z s3-bucket: gha-artifacts 2022-09-27T15:28:19.6204767Z region: us-east-1 2022-09-27T15:28:19.6204933Z env: 2022-09-27T15:28:19.6205196Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:28:19.6205386Z ##[endgroup] 2022-09-27T15:28:20.2413458Z Found 1 objects with prefix pytorch/pytorch/3133193930/linux-focal-py3.7-gcc7/ 2022-09-27T15:28:20.2413923Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-09-27T15:28:23.2533404Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-09-27T15:28:23.2533662Z 2022-09-27T15:28:23.2534405Z Artifact download has finished successfully 2022-09-27T15:28:23.2671640Z ##[group]Run unzip -o artifacts.zip 2022-09-27T15:28:23.2671860Z unzip -o artifacts.zip 2022-09-27T15:28:23.2683247Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T15:28:23.2683451Z env: 2022-09-27T15:28:23.2683617Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:28:23.2683795Z ##[endgroup] 2022-09-27T15:28:23.2741738Z Archive: artifacts.zip 2022-09-27T15:28:23.2743080Z creating: dist/ 2022-09-27T15:28:24.0318502Z inflating: dist/torch-1.13.0a0+git52424e2-cp37-cp37m-linux_x86_64.whl 2022-09-27T15:28:24.0319121Z creating: build/custom_test_artifacts/ 2022-09-27T15:28:24.0319674Z creating: build/custom_test_artifacts/custom-op-build/ 2022-09-27T15:28:24.0320295Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2022-09-27T15:28:24.0322211Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log 2022-09-27T15:28:24.0322946Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/ 2022-09-27T15:28:24.0323682Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-09-27T15:28:24.0324447Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-09-27T15:28:24.0325174Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-09-27T15:28:24.0326429Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-09-27T15:28:24.0328454Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-09-27T15:28:24.0329242Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-09-27T15:28:24.0329992Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-09-27T15:28:24.0331615Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-09-27T15:28:24.0333742Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-09-27T15:28:24.0335294Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-09-27T15:28:24.0336154Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-09-27T15:28:24.0338334Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-09-27T15:28:24.0339206Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-09-27T15:28:24.0339976Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2022-09-27T15:28:24.0340706Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log 2022-09-27T15:28:24.0341478Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2022-09-27T15:28:24.0342194Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2022-09-27T15:28:24.0343002Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2022-09-27T15:28:24.0343838Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2022-09-27T15:28:24.0344662Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2022-09-27T15:28:24.0345618Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2022-09-27T15:28:24.0346448Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2022-09-27T15:28:24.0347229Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2022-09-27T15:28:24.0348023Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2022-09-27T15:28:24.0348811Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2022-09-27T15:28:24.0349571Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2022-09-27T15:28:24.0363601Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2022-09-27T15:28:24.0451257Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2022-09-27T15:28:24.0452094Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2022-09-27T15:28:24.0452934Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2022-09-27T15:28:24.0453820Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2022-09-27T15:28:24.0454673Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2022-09-27T15:28:24.0455516Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2022-09-27T15:28:24.0456370Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2022-09-27T15:28:24.0457198Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2022-09-27T15:28:24.0458037Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2022-09-27T15:28:24.0458848Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2022-09-27T15:28:24.0459644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2022-09-27T15:28:24.0473650Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2022-09-27T15:28:24.0536512Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2022-09-27T15:28:24.0537438Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-09-27T15:28:24.0538285Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2022-09-27T15:28:24.0539058Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2022-09-27T15:28:24.0539936Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2022-09-27T15:28:24.0540691Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2022-09-27T15:28:24.0541557Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2022-09-27T15:28:24.0542595Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2022-09-27T15:28:24.0543295Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2022-09-27T15:28:24.0614942Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2022-09-27T15:28:24.0662920Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2022-09-27T15:28:24.0663586Z creating: build/custom_test_artifacts/jit-hook-build/ 2022-09-27T15:28:24.0664201Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2022-09-27T15:28:24.0666510Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log 2022-09-27T15:28:24.0667221Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/ 2022-09-27T15:28:24.0668110Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-09-27T15:28:24.0668856Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-09-27T15:28:24.0669574Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-09-27T15:28:24.0670657Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-09-27T15:28:24.0672652Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-09-27T15:28:24.0673427Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-09-27T15:28:24.0674185Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-09-27T15:28:24.0675933Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-09-27T15:28:24.0677015Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-09-27T15:28:24.0678489Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-09-27T15:28:24.0679016Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-09-27T15:28:24.0680356Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-09-27T15:28:24.0680925Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-09-27T15:28:24.0681411Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2022-09-27T15:28:24.0681948Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log 2022-09-27T15:28:24.0682435Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2022-09-27T15:28:24.0682892Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2022-09-27T15:28:24.0683430Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2022-09-27T15:28:24.0683986Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2022-09-27T15:28:24.0684494Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2022-09-27T15:28:24.0684936Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2022-09-27T15:28:24.0685489Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2022-09-27T15:28:24.0685992Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2022-09-27T15:28:24.0686489Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2022-09-27T15:28:24.0687103Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2022-09-27T15:28:24.0687562Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2022-09-27T15:28:24.0701961Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2022-09-27T15:28:24.0750569Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2022-09-27T15:28:24.0751147Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-09-27T15:28:24.0751732Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2022-09-27T15:28:24.0752188Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2022-09-27T15:28:24.0752590Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2022-09-27T15:28:24.0753119Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2022-09-27T15:28:24.0754643Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2022-09-27T15:28:24.0755148Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2022-09-27T15:28:24.0755650Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2022-09-27T15:28:24.0793404Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2022-09-27T15:28:24.0793772Z creating: build/custom_test_artifacts/custom-backend-build/ 2022-09-27T15:28:24.0794148Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2022-09-27T15:28:24.0796399Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log 2022-09-27T15:28:24.0796834Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/ 2022-09-27T15:28:24.0797274Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-09-27T15:28:24.0797718Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-09-27T15:28:24.0798139Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-09-27T15:28:24.0799458Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-09-27T15:28:24.0800823Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-09-27T15:28:24.0801270Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-09-27T15:28:24.0801692Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-09-27T15:28:24.0803181Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-09-27T15:28:24.0804449Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-09-27T15:28:24.0805781Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-09-27T15:28:24.0806254Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-09-27T15:28:24.0807654Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-09-27T15:28:24.0808645Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-09-27T15:28:24.0809278Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2022-09-27T15:28:24.0809848Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log 2022-09-27T15:28:24.0810473Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2022-09-27T15:28:24.0811185Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2022-09-27T15:28:24.0811769Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2022-09-27T15:28:24.0812412Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2022-09-27T15:28:24.0813024Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2022-09-27T15:28:24.0813617Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2022-09-27T15:28:24.0814090Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2022-09-27T15:28:24.0814565Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2022-09-27T15:28:24.0815038Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2022-09-27T15:28:24.0815573Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2022-09-27T15:28:24.0816056Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2022-09-27T15:28:24.0816974Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2022-09-27T15:28:24.0930468Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2022-09-27T15:28:24.0931092Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2022-09-27T15:28:24.0931726Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2022-09-27T15:28:24.0932327Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2022-09-27T15:28:24.0932947Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2022-09-27T15:28:24.0933577Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2022-09-27T15:28:24.0934197Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2022-09-27T15:28:24.0934680Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2022-09-27T15:28:24.0935163Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2022-09-27T15:28:24.0935634Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2022-09-27T15:28:24.0936108Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2022-09-27T15:28:24.0949863Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2022-09-27T15:28:24.0994108Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2022-09-27T15:28:24.0994827Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-09-27T15:28:24.0995434Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2022-09-27T15:28:24.0995916Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2022-09-27T15:28:24.0996462Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2022-09-27T15:28:24.0997267Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2022-09-27T15:28:24.0998585Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2022-09-27T15:28:24.0999457Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2022-09-27T15:28:24.0999932Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2022-09-27T15:28:24.1091151Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2022-09-27T15:28:24.1125936Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2022-09-27T15:28:24.1126628Z creating: build/lib/ 2022-09-27T15:28:24.1127242Z inflating: build/lib/libclog.a 2022-09-27T15:28:24.1178542Z inflating: build/lib/libgtest.a 2022-09-27T15:28:24.1186317Z inflating: build/lib/libpthreadpool.a 2022-09-27T15:28:24.1257496Z inflating: build/lib/libbenchmark.a 2022-09-27T15:28:24.1339886Z inflating: build/lib/libprotobuf-lite.a 2022-09-27T15:28:24.1346597Z inflating: build/lib/libittnotify.a 2022-09-27T15:28:24.1371488Z inflating: build/lib/libtensorpipe_uv.a 2022-09-27T15:28:24.1431273Z inflating: build/lib/libasmjit.a 2022-09-27T15:28:24.1521492Z inflating: build/lib/libgloo.a 2022-09-27T15:28:24.1936769Z inflating: build/lib/libprotobuf.a 2022-09-27T15:28:24.1951485Z inflating: build/lib/libfmt.a 2022-09-27T15:28:24.1952059Z inflating: build/lib/libfoxi_loader.a 2022-09-27T15:28:24.2008462Z inflating: build/lib/libc10.so 2022-09-27T15:28:24.2009606Z inflating: build/lib/libtorch_global_deps.so 2022-09-27T15:28:24.2016855Z inflating: build/lib/libcpuinfo.a 2022-09-27T15:28:24.2023882Z inflating: build/lib/libcpuinfo_internals.a 2022-09-27T15:28:24.2036313Z inflating: build/lib/libqnnpack.a 2022-09-27T15:28:24.2054839Z inflating: build/lib/libpytorch_qnnpack.a 2022-09-27T15:28:24.2056701Z inflating: build/lib/libnnpack_reference_layers.a 2022-09-27T15:28:24.2501512Z inflating: build/lib/libprotoc.a 2022-09-27T15:28:24.2516025Z inflating: build/lib/libgmock.a 2022-09-27T15:28:24.2516838Z inflating: build/lib/libgtest_main.a 2022-09-27T15:28:24.2517190Z inflating: build/lib/libbenchmark_main.a 2022-09-27T15:28:24.8782434Z inflating: build/lib/libdnnl.a 2022-09-27T15:28:24.8799410Z inflating: build/lib/libnnpack.a 2022-09-27T15:28:24.9305713Z inflating: build/lib/libtensorpipe.a 2022-09-27T15:28:24.9306210Z inflating: build/lib/libgmock_main.a 2022-09-27T15:28:25.0482779Z inflating: build/lib/libfbgemm.a 2022-09-27T15:28:25.1356321Z inflating: build/lib/libdnnl_graph.a 2022-09-27T15:28:25.1546917Z inflating: build/lib/libkineto.a 2022-09-27T15:28:25.1582369Z inflating: build/lib/libcaffe2_protos.a 2022-09-27T15:28:25.1619414Z inflating: build/lib/libonnx_proto.a 2022-09-27T15:28:25.1730312Z inflating: build/lib/libXNNPACK.a 2022-09-27T15:28:25.2255982Z inflating: build/lib/libonnx.a 2022-09-27T15:28:27.0221547Z inflating: build/lib/libtorch_cpu.so 2022-09-27T15:28:27.0222008Z inflating: build/lib/libtorch.so 2022-09-27T15:28:27.0240892Z inflating: build/lib/libjitbackend_test.so 2022-09-27T15:28:27.0287740Z inflating: build/lib/libtorchbind_test.so 2022-09-27T15:28:27.0311764Z inflating: build/lib/libbackend_with_compiler.so 2022-09-27T15:28:27.0315419Z inflating: build/lib/libshm.so 2022-09-27T15:28:27.1594496Z inflating: build/lib/libtorch_python.so 2022-09-27T15:28:27.1625415Z inflating: build/lib/libnnapi_backend.so 2022-09-27T15:28:27.1625829Z creating: build/bin/ 2022-09-27T15:28:27.1666986Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2022-09-27T15:28:27.1710303Z inflating: build/bin/c10_DeviceGuard_test 2022-09-27T15:28:27.1752359Z inflating: build/bin/c10_Device_test 2022-09-27T15:28:27.1801894Z inflating: build/bin/c10_DispatchKeySet_test 2022-09-27T15:28:27.1842012Z inflating: build/bin/c10_StreamGuard_test 2022-09-27T15:28:27.1883951Z inflating: build/bin/c10_SymInt_test 2022-09-27T15:28:27.1931076Z inflating: build/bin/c10_InlineDeviceGuard_test 2022-09-27T15:28:27.1977577Z inflating: build/bin/c10_InlineStreamGuard_test 2022-09-27T15:28:27.2025621Z inflating: build/bin/c10_SizesAndStrides_test 2022-09-27T15:28:27.2065438Z inflating: build/bin/c10_Array_test 2022-09-27T15:28:27.2109823Z inflating: build/bin/c10_Bitset_test 2022-09-27T15:28:27.2150394Z inflating: build/bin/c10_ConstexprCrc_test 2022-09-27T15:28:27.2193217Z inflating: build/bin/c10_C++17_test 2022-09-27T15:28:27.2234244Z inflating: build/bin/c10_DeadlockDetection_test 2022-09-27T15:28:27.2281682Z inflating: build/bin/c10_LeftRight_test 2022-09-27T15:28:27.2323417Z inflating: build/bin/c10_Half_test 2022-09-27T15:28:27.2376029Z inflating: build/bin/c10_Metaprogramming_test 2022-09-27T15:28:27.2418168Z inflating: build/bin/c10_Synchronized_test 2022-09-27T15:28:27.2466157Z inflating: build/bin/c10_ThreadLocal_test 2022-09-27T15:28:27.2586590Z inflating: build/bin/c10_SmallVectorTest 2022-09-27T15:28:27.2630487Z inflating: build/bin/c10_TypeIndex_test 2022-09-27T15:28:27.2672779Z inflating: build/bin/c10_TypeList_test 2022-09-27T15:28:27.2712948Z inflating: build/bin/c10_TypeTraits_test 2022-09-27T15:28:27.2756218Z inflating: build/bin/c10_accumulate_test 2022-09-27T15:28:27.2802977Z inflating: build/bin/c10_bfloat16_test 2022-09-27T15:28:27.2848015Z inflating: build/bin/c10_complex_math_test 2022-09-27T15:28:27.2894665Z inflating: build/bin/c10_complex_test 2022-09-27T15:28:27.2986387Z inflating: build/bin/c10_either_test 2022-09-27T15:28:27.3030158Z inflating: build/bin/c10_exception_test 2022-09-27T15:28:27.3072196Z inflating: build/bin/c10_flags_test 2022-09-27T15:28:27.3114794Z inflating: build/bin/c10_irange_test 2022-09-27T15:28:27.3255999Z inflating: build/bin/c10_intrusive_ptr_test 2022-09-27T15:28:27.3304338Z inflating: build/bin/c10_logging_test 2022-09-27T15:28:27.3367286Z inflating: build/bin/c10_optional_test 2022-09-27T15:28:27.3419341Z inflating: build/bin/c10_ordered_preserving_dict_test 2022-09-27T15:28:27.3465416Z inflating: build/bin/c10_registry_test 2022-09-27T15:28:27.3514982Z inflating: build/bin/c10_string_view_test 2022-09-27T15:28:27.3557902Z inflating: build/bin/c10_tempfile_test 2022-09-27T15:28:27.3605197Z inflating: build/bin/c10_typeid_test 2022-09-27T15:28:27.3652680Z inflating: build/bin/c10_intrusive_ptr_benchmark 2022-09-27T15:28:27.4064383Z inflating: build/bin/protoc-3.13.0.0 2022-09-27T15:28:27.4476370Z inflating: build/bin/protoc 2022-09-27T15:28:27.4752554Z inflating: build/bin/vec_test_all_types_AVX2 2022-09-27T15:28:27.4999511Z inflating: build/bin/vec_test_all_types_DEFAULT 2022-09-27T15:28:27.5044748Z inflating: build/bin/FileStoreTest 2022-09-27T15:28:27.5090056Z inflating: build/bin/HashStoreTest 2022-09-27T15:28:27.5141244Z inflating: build/bin/TCPStoreTest 2022-09-27T15:28:27.5143853Z inflating: build/bin/example_allreduce 2022-09-27T15:28:27.5199498Z inflating: build/bin/ProcessGroupGlooTest 2022-09-27T15:28:27.5243817Z inflating: build/bin/Dimname_test 2022-09-27T15:28:27.5305477Z inflating: build/bin/Dict_test 2022-09-27T15:28:27.5359923Z inflating: build/bin/MaybeOwned_test 2022-09-27T15:28:27.5408301Z inflating: build/bin/NamedTensor_test 2022-09-27T15:28:27.5458941Z inflating: build/bin/apply_utils_test 2022-09-27T15:28:27.5510399Z inflating: build/bin/basic 2022-09-27T15:28:27.5560393Z inflating: build/bin/atest 2022-09-27T15:28:27.5606106Z inflating: build/bin/broadcast_test 2022-09-27T15:28:27.5656164Z inflating: build/bin/cpu_generator_test 2022-09-27T15:28:27.5700715Z inflating: build/bin/cpu_profiling_allocator_test 2022-09-27T15:28:27.5748603Z inflating: build/bin/dispatch_key_set_test 2022-09-27T15:28:27.5790237Z inflating: build/bin/dlconvertor_test 2022-09-27T15:28:27.5864125Z inflating: build/bin/cpu_rng_test 2022-09-27T15:28:27.5913340Z inflating: build/bin/extension_backend_test 2022-09-27T15:28:27.5960097Z inflating: build/bin/half_test 2022-09-27T15:28:27.6039063Z inflating: build/bin/ivalue_test 2022-09-27T15:28:27.6080638Z inflating: build/bin/lazy_tensor_test 2022-09-27T15:28:27.6124489Z inflating: build/bin/math_kernel_test 2022-09-27T15:28:27.6168251Z inflating: build/bin/memory_format_test 2022-09-27T15:28:27.6212265Z inflating: build/bin/memory_overlapping_test 2022-09-27T15:28:27.6254775Z inflating: build/bin/operator_name_test 2022-09-27T15:28:27.6301822Z inflating: build/bin/native_test 2022-09-27T15:28:27.6345503Z inflating: build/bin/mobile_memory_cleanup 2022-09-27T15:28:27.6387288Z inflating: build/bin/operators_test 2022-09-27T15:28:27.6431394Z inflating: build/bin/packedtensoraccessor_test 2022-09-27T15:28:27.6479715Z inflating: build/bin/quantized_test 2022-09-27T15:28:27.6534071Z inflating: build/bin/pow_test 2022-09-27T15:28:27.6576748Z inflating: build/bin/reportMemoryUsage_test 2022-09-27T15:28:27.6618321Z inflating: build/bin/reduce_ops_test 2022-09-27T15:28:27.6665483Z inflating: build/bin/scalar_tensor_test 2022-09-27T15:28:27.6713538Z inflating: build/bin/scalar_test 2022-09-27T15:28:27.6756677Z inflating: build/bin/stride_properties_test 2022-09-27T15:28:27.6822040Z inflating: build/bin/tensor_iterator_test 2022-09-27T15:28:27.6868817Z inflating: build/bin/type_ptr_test 2022-09-27T15:28:27.6870834Z inflating: build/bin/thread_init_test 2022-09-27T15:28:27.6912333Z inflating: build/bin/variant_test 2022-09-27T15:28:27.6959375Z inflating: build/bin/test_parallel 2022-09-27T15:28:27.7002867Z inflating: build/bin/undefined_tensor_test 2022-09-27T15:28:27.7053747Z inflating: build/bin/type_test 2022-09-27T15:28:27.7054898Z inflating: build/bin/verify_api_visibility 2022-09-27T15:28:27.7113083Z inflating: build/bin/vmap_test 2022-09-27T15:28:27.7155792Z inflating: build/bin/weakref_test 2022-09-27T15:28:27.7198495Z inflating: build/bin/wrapdim_test 2022-09-27T15:28:27.7290358Z inflating: build/bin/List_test 2022-09-27T15:28:27.7331208Z inflating: build/bin/xla_tensor_test 2022-09-27T15:28:27.7381431Z inflating: build/bin/IListRef_test 2022-09-27T15:28:27.7484487Z inflating: build/bin/kernel_function_legacy_test 2022-09-27T15:28:27.7565583Z inflating: build/bin/kernel_function_test 2022-09-27T15:28:27.7619658Z inflating: build/bin/KernelFunction_test 2022-09-27T15:28:27.7728314Z inflating: build/bin/kernel_lambda_legacy_test 2022-09-27T15:28:27.7815577Z inflating: build/bin/kernel_lambda_test 2022-09-27T15:28:27.7865947Z inflating: build/bin/kernel_stackbased_test 2022-09-27T15:28:27.7907989Z inflating: build/bin/CppSignature_test 2022-09-27T15:28:27.7988997Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2022-09-27T15:28:27.8029119Z inflating: build/bin/op_allowlist_test 2022-09-27T15:28:27.8073666Z inflating: build/bin/inline_container_test 2022-09-27T15:28:27.8317531Z inflating: build/bin/op_registration_test 2022-09-27T15:28:27.8364731Z inflating: build/bin/backend_fallback_test 2022-09-27T15:28:27.8378086Z inflating: build/bin/tutorial_tensorexpr 2022-09-27T15:28:27.8423231Z inflating: build/bin/test_dist_autograd 2022-09-27T15:28:27.8425308Z inflating: build/bin/parallel_benchmark 2022-09-27T15:28:27.8484111Z inflating: build/bin/test_cpp_rpc 2022-09-27T15:28:27.8542007Z inflating: build/bin/test_mobile_nnc 2022-09-27T15:28:27.8550657Z inflating: build/bin/aot_model_compiler_test 2022-09-27T15:28:27.8844571Z inflating: build/bin/test_lazy 2022-09-27T15:28:27.9523157Z inflating: build/bin/test_tensorexpr 2022-09-27T15:28:27.9527446Z inflating: build/bin/torch_shm_manager 2022-09-27T15:28:28.0530939Z inflating: build/bin/test_api 2022-09-27T15:28:28.1026070Z inflating: build/bin/test_jit 2022-09-27T15:28:28.1027330Z inflating: .pytorch-test-times.json 2022-09-27T15:28:28.1049886Z ##[group]Run df -H 2022-09-27T15:28:28.1050070Z df -H 2022-09-27T15:28:28.1062124Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T15:28:28.1062436Z env: 2022-09-27T15:28:28.1062614Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:28:28.1062788Z ##[endgroup] 2022-09-27T15:28:28.1113074Z Filesystem Size Used Avail Use% Mounted on 2022-09-27T15:28:28.1113450Z devtmpfs 8.2G 0 8.2G 0% /dev 2022-09-27T15:28:28.1113801Z tmpfs 8.2G 443k 8.2G 1% /dev/shm 2022-09-27T15:28:28.1114105Z tmpfs 8.2G 431k 8.2G 1% /run 2022-09-27T15:28:28.1114327Z tmpfs 8.2G 0 8.2G 0% /sys/fs/cgroup 2022-09-27T15:28:28.1114716Z /dev/nvme0n1p1 162G 16G 146G 10% / 2022-09-27T15:28:28.1150719Z ##[group]Run .github/scripts/parse_ref.py 2022-09-27T15:28:28.1150965Z .github/scripts/parse_ref.py 2022-09-27T15:28:28.1161728Z shell: /usr/bin/bash -e {0} 2022-09-27T15:28:28.1161901Z env: 2022-09-27T15:28:28.1162079Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:28:28.1162272Z ##[endgroup] 2022-09-27T15:28:28.1391272Z ##[group]Run set -x 2022-09-27T15:28:28.1391625Z set -x 2022-09-27T15:28:28.1391782Z  2022-09-27T15:28:28.1392076Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2022-09-27T15:28:28.1392341Z  TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh 2022-09-27T15:28:28.1392590Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2022-09-27T15:28:28.1392832Z  TEST_COMMAND=.jenkins/caffe2/test.sh 2022-09-27T15:28:28.1393034Z else 2022-09-27T15:28:28.1393230Z  TEST_COMMAND=.jenkins/pytorch/test.sh 2022-09-27T15:28:28.1393431Z fi 2022-09-27T15:28:28.1393591Z  2022-09-27T15:28:28.1393817Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}") 2022-09-27T15:28:28.1394057Z  2022-09-27T15:28:28.1394277Z # sanitize the input commit message and PR body here: 2022-09-27T15:28:28.1394494Z # 2022-09-27T15:28:28.1395085Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment 2022-09-27T15:28:28.1395464Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028 2022-09-27T15:28:28.1395768Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}" 2022-09-27T15:28:28.1396000Z PR_BODY="${PR_BODY//[$'\n\r']}" 2022-09-27T15:28:28.1396188Z  2022-09-27T15:28:28.1396437Z # then trim all special characters like single and double quotes to avoid unescaped inputs to 2022-09-27T15:28:28.1396718Z # wreak havoc internally 2022-09-27T15:28:28.1396959Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}" 2022-09-27T15:28:28.1397201Z export PR_BODY="${PR_BODY//[\'\"]}" 2022-09-27T15:28:28.1397379Z  2022-09-27T15:28:28.1397609Z # detached container should get cleaned up by teardown_ec2_linux 2022-09-27T15:28:28.1397905Z # TODO: Stop building test binaries as part of the build phase 2022-09-27T15:28:28.1398167Z # Used for GPU_FLAG since that doesn't play nice 2022-09-27T15:28:28.1398409Z # shellcheck disable=SC2086,SC2090 2022-09-27T15:28:28.1398759Z container_name=$(docker run \ 2022-09-27T15:28:28.1399074Z  ${GPU_FLAG:-} \ 2022-09-27T15:28:28.1399305Z  -e BUILD_ENVIRONMENT \ 2022-09-27T15:28:28.1399500Z  -e PR_NUMBER \ 2022-09-27T15:28:28.1399692Z  -e GITHUB_ACTIONS \ 2022-09-27T15:28:28.1399866Z  -e BASE_SHA \ 2022-09-27T15:28:28.1400045Z  -e BRANCH \ 2022-09-27T15:28:28.1400218Z  -e SHA1 \ 2022-09-27T15:28:28.1400394Z  -e AWS_DEFAULT_REGION \ 2022-09-27T15:28:28.1400588Z  -e IN_WHEEL_TEST \ 2022-09-27T15:28:28.1400777Z  -e SHARD_NUMBER \ 2022-09-27T15:28:28.1400950Z  -e TEST_CONFIG \ 2022-09-27T15:28:28.1401141Z  -e NUM_TEST_SHARDS \ 2022-09-27T15:28:28.1401326Z  -e PR_BODY \ 2022-09-27T15:28:28.1401505Z  -e COMMIT_MESSAGES \ 2022-09-27T15:28:28.1401713Z  -e PYTORCH_RETRY_TEST_CASES \ 2022-09-27T15:28:28.1401948Z  -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \ 2022-09-27T15:28:28.1402144Z  -e PR_LABELS \ 2022-09-27T15:28:28.1402357Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2022-09-27T15:28:28.1402566Z  -e SCCACHE_BUCKET \ 2022-09-27T15:28:28.1402769Z  -e SCCACHE_S3_KEY_PREFIX \ 2022-09-27T15:28:28.1402953Z  -e XLA_CUDA \ 2022-09-27T15:28:28.1403162Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2022-09-27T15:28:28.1403412Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2022-09-27T15:28:28.1403636Z  --ulimit stack=10485760:83886080 \ 2022-09-27T15:28:28.1403864Z  --security-opt seccomp=unconfined \ 2022-09-27T15:28:28.1404088Z  --cap-add=SYS_PTRACE \ 2022-09-27T15:28:28.1404268Z  --ipc=host \ 2022-09-27T15:28:28.1404551Z  --shm-size="${SHM_SIZE}" \ 2022-09-27T15:28:28.1404742Z  --tty \ 2022-09-27T15:28:28.1404903Z  --detach \ 2022-09-27T15:28:28.1405186Z  --name="${container_name}" \ 2022-09-27T15:28:28.1405386Z  --user jenkins \ 2022-09-27T15:28:28.1405621Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2022-09-27T15:28:28.1405863Z  -w /var/lib/jenkins/workspace \ 2022-09-27T15:28:28.1406071Z  "${DOCKER_IMAGE}" 2022-09-27T15:28:28.1406244Z ) 2022-09-27T15:28:28.1406728Z docker exec -t "${container_name}" sh -c "pip install dist/*.whl && ${TEST_COMMAND}" 2022-09-27T15:28:28.1417365Z shell: /usr/bin/bash -e {0} 2022-09-27T15:28:28.1417553Z env: 2022-09-27T15:28:28.1417715Z GIT_DEFAULT_BRANCH: master 2022-09-27T15:28:28.1417948Z BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7 2022-09-27T15:28:28.1418161Z PR_NUMBER: 85462 2022-09-27T15:28:28.1418326Z BRANCH: pull/85462 2022-09-27T15:28:28.1418542Z SHA1: 52424e2bf38e454d535881fed9628d3e20f4f944 2022-09-27T15:28:28.1418776Z BASE_SHA: 76d60778eb01b4213735be1c6e126fe2da519b8e 2022-09-27T15:28:28.1418980Z PYTORCH_RETRY_TEST_CASES: 1 2022-09-27T15:28:28.1419193Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-09-27T15:28:28.1419394Z TEST_CONFIG: distributed 2022-09-27T15:28:28.1419573Z SHARD_NUMBER: 1 2022-09-27T15:28:28.1419731Z NUM_TEST_SHARDS: 2 2022-09-27T15:28:28.1421441Z PR_BODY: Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory. I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug: ``` import torch import weakref from transformers import DetrForObjectDetection class X: def __init__(self, optimizer): self.optimizer = optimizer # Will cause cyclic reference. self.func = self.dummy # Will work as expected, memory cleared after instance count is zero. # self.func = weakref.WeakMethod(self.dummy) def dummy(self, x): return 1. def test(): model = DetrForObjectDetection.from_pretrained('facebook/detr-resnet-50') model.to('cuda') optimizer = torch.optim.Adam(model.parameters()) x = X(optimizer) test() print(f'{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()}') # Should print (, 0), but with cyclic reference, it will print (, ). ``` 2022-09-27T15:28:28.1423317Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-09-27T15:28:28.1423549Z SCCACHE_S3_KEY_PREFIX: pull 2022-09-27T15:28:28.1423734Z SHM_SIZE: 1g 2022-09-27T15:28:28.1424077Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29 2022-09-27T15:28:28.1424392Z XLA_CUDA: 2022-09-27T15:28:28.1424649Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-09-27T15:28:28.1424908Z ##[endgroup] 2022-09-27T15:28:28.1449364Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2022-09-27T15:28:28.1449748Z + [[ linux-focal-py3.7-gcc7 == *onnx* ]] 2022-09-27T15:28:28.1449986Z + TEST_COMMAND=.jenkins/pytorch/test.sh 2022-09-27T15:28:28.1452604Z ++ git cherry -v origin/master 2022-09-27T15:28:28.1648513Z + COMMIT_MESSAGES='+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR 2022-09-27T15:28:28.1649272Z + acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR 2022-09-27T15:28:28.1649580Z + 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form 2022-09-27T15:28:28.1649936Z + 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference 2022-09-27T15:28:28.1650208Z + 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting' 2022-09-27T15:28:28.1651288Z + COMMIT_MESSAGES='+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting' 2022-09-27T15:28:28.1664429Z + PR_BODY='Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X: def __init__(self, optimizer): self.optimizer = optimizer # Will cause cyclic reference. self.func = self.dummy # Will work as expected, memory cleared after instance count is zero. # self.func = weakref.WeakMethod(self.dummy) def dummy(self, x): return 1.def test(): model = DetrForObjectDetection.from_pretrained('\''facebook/detr-resnet-50'\'') model.to('\''cuda'\'') optimizer = torch.optim.Adam(model.parameters()) x = X(optimizer)test()print(f'\''{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()}'\'') # Should print (, 0), but with cyclic reference, it will print (, ).```' 2022-09-27T15:28:28.1666922Z + export 'COMMIT_MESSAGES=+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting' 2022-09-27T15:28:28.1668011Z + COMMIT_MESSAGES='+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting' 2022-09-27T15:28:28.1674268Z + export 'PR_BODY=Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X: def __init__(self, optimizer): self.optimizer = optimizer # Will cause cyclic reference. self.func = self.dummy # Will work as expected, memory cleared after instance count is zero. # self.func = weakref.WeakMethod(self.dummy) def dummy(self, x): return 1.def test(): model = DetrForObjectDetection.from_pretrained(facebook/detr-resnet-50) model.to(cuda) optimizer = torch.optim.Adam(model.parameters()) x = X(optimizer)test()print(f{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()}) # Should print (, 0), but with cyclic reference, it will print (, ).```' 2022-09-27T15:28:28.1678754Z + PR_BODY='Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X: def __init__(self, optimizer): self.optimizer = optimizer # Will cause cyclic reference. self.func = self.dummy # Will work as expected, memory cleared after instance count is zero. # self.func = weakref.WeakMethod(self.dummy) def dummy(self, x): return 1.def test(): model = DetrForObjectDetection.from_pretrained(facebook/detr-resnet-50) model.to(cuda) optimizer = torch.optim.Adam(model.parameters()) x = X(optimizer)test()print(f{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()}) # Should print (, 0), but with cyclic reference, it will print (, ).```' 2022-09-27T15:28:28.1680702Z +++ nproc --ignore=2 2022-09-27T15:28:28.1703720Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_3133193930 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29 2022-09-27T15:28:39.0560373Z + container_name=51c56aedd00ae5cc22b203da5b2416a90eefb59e9fc34e33dc8868914cc83ee3 2022-09-27T15:28:39.0561199Z + docker exec -t 51c56aedd00ae5cc22b203da5b2416a90eefb59e9fc34e33dc8868914cc83ee3 sh -c 'pip install dist/*.whl && .jenkins/pytorch/test.sh' 2022-09-27T15:28:39.4892800Z Processing ./dist/torch-1.13.0a0+git52424e2-cp37-cp37m-linux_x86_64.whl 2022-09-27T15:28:39.5644195Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.13.0a0+git52424e2) (4.3.0) 2022-09-27T15:28:40.3519986Z Installing collected packages: torch 2022-09-27T15:28:46.5148952Z Successfully installed torch-1.13.0a0+git52424e2 2022-09-27T15:28:46.6507876Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2022-09-27T15:28:46.6664361Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.7/site-packages/torch 2022-09-27T15:28:46.6664981Z + TORCH_BIN_DIR=/opt/conda/lib/python3.7/site-packages/torch/bin 2022-09-27T15:28:46.6665449Z + TORCH_LIB_DIR=/opt/conda/lib/python3.7/site-packages/torch/lib 2022-09-27T15:28:46.6666041Z + TORCH_TEST_DIR=/opt/conda/lib/python3.7/site-packages/torch/test 2022-09-27T15:28:46.6666411Z + BUILD_DIR=build 2022-09-27T15:28:46.6666731Z + BUILD_RENAMED_DIR=build_renamed 2022-09-27T15:28:46.6667050Z + BUILD_BIN_DIR=build/bin 2022-09-27T15:28:46.6667323Z + export VALGRIND=ON 2022-09-27T15:28:46.6667597Z + VALGRIND=ON 2022-09-27T15:28:46.6667994Z + [[ linux-focal-py3.7-gcc7 == *clang9* ]] 2022-09-27T15:28:46.6668389Z + [[ linux-focal-py3.7-gcc7 != *bazel* ]] 2022-09-27T15:28:46.6669047Z ++ realpath build/custom_test_artifacts 2022-09-27T15:28:46.6676498Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2022-09-27T15:28:46.6678837Z ++ dirname .jenkins/pytorch/test.sh 2022-09-27T15:28:46.6684488Z + source .jenkins/pytorch/common.sh 2022-09-27T15:28:46.6687860Z +++ dirname .jenkins/pytorch/common.sh 2022-09-27T15:28:46.6692741Z ++ source .jenkins/pytorch/common_utils.sh 2022-09-27T15:28:46.6694730Z +++ declare -f -t trap_add 2022-09-27T15:28:46.6698860Z ++ set -ex 2022-09-27T15:28:46.6699576Z ++ [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-09-27T15:28:46.6699873Z ++ BUILD_TEST_LIBTORCH=0 2022-09-27T15:28:46.6700125Z ++ [[ distributed == *xla* ]] 2022-09-27T15:28:46.6700606Z ++ [[ linux-focal-py3.7-gcc7 == *centos* ]] 2022-09-27T15:28:46.6701109Z ++ [[ linux-focal-py3.7-gcc7 == *linux-bionic* ]] 2022-09-27T15:28:46.6701406Z ++ [[ linux-focal-py3.7-gcc7 == *linux-focal* ]] 2022-09-27T15:28:46.6701651Z ++ which conda 2022-09-27T15:28:46.6709393Z /opt/conda/bin/conda 2022-09-27T15:28:46.6710987Z ++ conda install -q -y cmake 2022-09-27T15:28:48.7755310Z Collecting package metadata (current_repodata.json): ...working... done 2022-09-27T15:28:49.2416358Z Solving environment: ...working... done 2022-09-27T15:28:49.3302928Z 2022-09-27T15:28:49.3303413Z ## Package Plan ## 2022-09-27T15:28:49.3303678Z 2022-09-27T15:28:49.3303864Z environment location: /opt/conda 2022-09-27T15:28:49.3304243Z 2022-09-27T15:28:49.3304372Z added / updated specs: 2022-09-27T15:28:49.3304705Z - cmake 2022-09-27T15:28:49.3304794Z 2022-09-27T15:28:49.3304800Z 2022-09-27T15:28:49.3304911Z The following packages will be downloaded: 2022-09-27T15:28:49.3305052Z 2022-09-27T15:28:49.3305149Z package | build 2022-09-27T15:28:49.3305427Z ---------------------------|----------------- 2022-09-27T15:28:49.3305728Z bzip2-1.0.8 | h7b6447c_0 78 KB 2022-09-27T15:28:49.3306026Z c-ares-1.18.1 | h7f8727e_0 114 KB 2022-09-27T15:28:49.3306341Z certifi-2022.9.14 | py37h06a4308_0 155 KB 2022-09-27T15:28:49.3306623Z cmake-3.22.1 | h1fce559_0 7.3 MB 2022-09-27T15:28:49.3306919Z conda-22.9.0 | py37h06a4308_0 878 KB 2022-09-27T15:28:49.3307201Z expat-2.4.4 | h295c915_0 169 KB 2022-09-27T15:28:49.3307482Z krb5-1.19.2 | hac12032_0 1.2 MB 2022-09-27T15:28:49.3307752Z libcurl-7.82.0 | h0b77cf5_0 342 KB 2022-09-27T15:28:49.3308042Z libedit-3.1.20210910 | h7f8727e_0 166 KB 2022-09-27T15:28:49.3308324Z libev-4.33 | h7f8727e_1 111 KB 2022-09-27T15:28:49.3308602Z libnghttp2-1.46.0 | hce63b2e_0 680 KB 2022-09-27T15:28:49.3308891Z libssh2-1.10.0 | h8f2d780_0 274 KB 2022-09-27T15:28:49.3309173Z libuv-1.40.0 | h7b6447c_0 736 KB 2022-09-27T15:28:49.3309677Z lz4-c-1.9.3 | h295c915_1 185 KB 2022-09-27T15:28:49.3309961Z rhash-1.4.1 | h3c74f83_1 203 KB 2022-09-27T15:28:49.3310336Z zstd-1.5.2 | ha4553b6_0 488 KB 2022-09-27T15:28:49.3310637Z ------------------------------------------------------------ 2022-09-27T15:28:49.3310872Z Total: 13.0 MB 2022-09-27T15:28:49.3310994Z 2022-09-27T15:28:49.3311109Z The following NEW packages will be INSTALLED: 2022-09-27T15:28:49.3311251Z 2022-09-27T15:28:49.3323974Z bzip2 pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0 2022-09-27T15:28:49.3324393Z c-ares pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0 2022-09-27T15:28:49.3324747Z cmake pkgs/main/linux-64::cmake-3.22.1-h1fce559_0 2022-09-27T15:28:49.3325100Z expat pkgs/main/linux-64::expat-2.4.4-h295c915_0 2022-09-27T15:28:49.3325455Z krb5 pkgs/main/linux-64::krb5-1.19.2-hac12032_0 2022-09-27T15:28:49.3325796Z libcurl pkgs/main/linux-64::libcurl-7.82.0-h0b77cf5_0 2022-09-27T15:28:49.3326179Z libedit pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0 2022-09-27T15:28:49.3326537Z libev pkgs/main/linux-64::libev-4.33-h7f8727e_1 2022-09-27T15:28:49.3326905Z libnghttp2 pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0 2022-09-27T15:28:49.3327263Z libssh2 pkgs/main/linux-64::libssh2-1.10.0-h8f2d780_0 2022-09-27T15:28:49.3327615Z libuv pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0 2022-09-27T15:28:49.3327962Z lz4-c pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1 2022-09-27T15:28:49.3328307Z rhash pkgs/main/linux-64::rhash-1.4.1-h3c74f83_1 2022-09-27T15:28:49.3328640Z zstd pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0 2022-09-27T15:28:49.3328786Z 2022-09-27T15:28:49.3328888Z The following packages will be UPDATED: 2022-09-27T15:28:49.3329026Z 2022-09-27T15:28:49.3329234Z certifi 2022.6.15-py37h06a4308_0 --> 2022.9.14-py37h06a4308_0 2022-09-27T15:28:49.3329569Z conda 4.14.0-py37h06a4308_0 --> 22.9.0-py37h06a4308_0 2022-09-27T15:28:49.3329718Z 2022-09-27T15:28:49.3329723Z 2022-09-27T15:28:50.4942075Z Preparing transaction: ...working... done 2022-09-27T15:28:50.9023500Z Verifying transaction: ...working... done 2022-09-27T15:28:51.9513181Z Executing transaction: ...working... done 2022-09-27T15:28:52.0806324Z Retrieving notices: ...working... done 2022-09-27T15:28:52.2822972Z ++ [[ linux-focal-py3.7-gcc7 == *centos* ]] 2022-09-27T15:28:52.2823422Z + echo 'Environment variables' 2022-09-27T15:28:52.2823765Z Environment variables 2022-09-27T15:28:52.2824048Z + env 2022-09-27T15:28:52.2830574Z INSTALLED_DB=yes 2022-09-27T15:28:52.2831558Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-09-27T15:28:52.2831947Z BUILD_ENVIRONMENT=linux-focal-py3.7-gcc7 2022-09-27T15:28:52.2832317Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2022-09-27T15:28:52.2832548Z HOSTNAME=51c56aedd00a 2022-09-27T15:28:52.2833196Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_516d4d80-ecdc-4837-a7a6-15c4a89ab68f 2022-09-27T15:28:52.2833672Z GITHUB_ACTION=__self 2022-09-27T15:28:52.2834027Z GITHUB_RUN_NUMBER=50832 2022-09-27T15:28:52.2834303Z TEST_CONFIG=distributed 2022-09-27T15:28:52.2834871Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2022-09-27T15:28:52.2835237Z GITHUB_TRIGGERING_ACTOR=albanD 2022-09-27T15:28:52.2835593Z GITHUB_REF_TYPE=branch 2022-09-27T15:28:52.2835953Z TORCH_CUDA_ARCH_LIST=Maxwell 2022-09-27T15:28:52.2836260Z BASE_SHA=76d60778eb01b4213735be1c6e126fe2da519b8e 2022-09-27T15:28:52.2836533Z XLA_CUDA= 2022-09-27T15:28:52.2837196Z *** 2022-09-27T15:28:52.2837451Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2022-09-27T15:28:52.2837721Z GITHUB_ACTIONS=true 2022-09-27T15:28:52.2837998Z SHA1=52424e2bf38e454d535881fed9628d3e20f4f944 2022-09-27T15:28:52.2838543Z GITHUB_SHA=1faa2af6dbb8dd899ab20874e9966185467c5883 2022-09-27T15:28:52.2842127Z UCC_HOME=/usr 2022-09-27T15:28:52.2842554Z GITHUB_REF=refs/pull/85462/merge 2022-09-27T15:28:52.2843044Z SHARD_NUMBER=1 2022-09-27T15:28:52.2843385Z GITHUB_REF_PROTECTED=false 2022-09-27T15:28:52.2843721Z HOME=/var/lib/jenkins 2022-09-27T15:28:52.2844060Z GITHUB_API_URL=https://api.github.com 2022-09-27T15:28:52.2844401Z INSTALLED_THRIFT= 2022-09-27T15:28:52.2844676Z UCX_COMMIT= 2022-09-27T15:28:52.2844996Z SCCACHE_S3_KEY_PREFIX=pull 2022-09-27T15:28:52.2845337Z NUM_TEST_SHARDS=2 2022-09-27T15:28:52.2845627Z UCX_HOME=/usr 2022-09-27T15:28:52.2845975Z PYTORCH_RETRY_TEST_CASES=1 2022-09-27T15:28:52.2846709Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_516d4d80-ecdc-4837-a7a6-15c4a89ab68f 2022-09-27T15:28:52.2847339Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2022-09-27T15:28:52.2847847Z GITHUB_EVENT_NAME=pull_request 2022-09-27T15:28:52.2848174Z GITHUB_RUN_ID=3133193930 2022-09-27T15:28:52.2848769Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_516d4d80-ecdc-4837-a7a6-15c4a89ab68f 2022-09-27T15:28:52.2849075Z GITHUB_ACTOR=kongzii 2022-09-27T15:28:52.2849251Z PR_NUMBER=85462 2022-09-27T15:28:52.2849423Z DESIRED_CUDA= 2022-09-27T15:28:52.2849582Z GITHUB_RUN_ATTEMPT=2 2022-09-27T15:28:52.2849754Z VALGRIND=ON 2022-09-27T15:28:52.2849975Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2022-09-27T15:28:52.2850188Z TERM=xterm 2022-09-27T15:28:52.2850356Z INSTALLED_VISION=yes 2022-09-27T15:28:52.2850534Z BRANCH=pull/85462 2022-09-27T15:28:52.2850707Z OPENSSL_ROOT_DIR=/opt/openssl 2022-09-27T15:28:52.2850898Z CUDA_PATH=/usr/local/cuda 2022-09-27T15:28:52.2851267Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2022-09-27T15:28:52.2851552Z GITHUB_SERVER_URL=https://github.com 2022-09-27T15:28:52.2851747Z UCC_COMMIT= 2022-09-27T15:28:52.2851914Z INSTALLED_ANDROID= 2022-09-27T15:28:52.2852070Z SHLVL=1 2022-09-27T15:28:52.2852222Z MAX_JOBS=6 2022-09-27T15:28:52.2852734Z COMMIT_MESSAGES=+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting 2022-09-27T15:28:52.2853594Z GITHUB_REF_NAME=85462/merge 2022-09-27T15:28:52.2854148Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2022-09-27T15:28:52.2854393Z GITHUB_JOB=test 2022-09-27T15:28:52.2854588Z GITHUB_REPOSITORY=pytorch/pytorch 2022-09-27T15:28:52.2854794Z GITHUB_RETENTION_DAYS=90 2022-09-27T15:28:52.2855018Z OPENSSL_DIR=/opt/openssl 2022-09-27T15:28:52.2855266Z GITHUB_ACTION_REPOSITORY= 2022-09-27T15:28:52.2855783Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-09-27T15:28:52.2856302Z GITHUB_BASE_REF=master 2022-09-27T15:28:52.2856609Z CI=true 2022-09-27T15:28:52.2856798Z GITHUB_REPOSITORY_OWNER=pytorch 2022-09-27T15:28:52.2857011Z INSTALLED_PROTOBUF=yes 2022-09-27T15:28:52.2857250Z GITHUB_HEAD_REF=cycliclr-memory-fix 2022-09-27T15:28:52.2857450Z GITHUB_ACTION_REF= 2022-09-27T15:28:52.2857722Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2022-09-27T15:28:52.2857938Z GITHUB_WORKFLOW=pull 2022-09-27T15:28:52.2858134Z DEBIAN_FRONTEND=noninteractive 2022-09-27T15:28:52.2861062Z PR_BODY=Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X: def __init__(self, optimizer): self.optimizer = optimizer # Will cause cyclic reference. self.func = self.dummy # Will work as expected, memory cleared after instance count is zero. # self.func = weakref.WeakMethod(self.dummy) def dummy(self, x): return 1.def test(): model = DetrForObjectDetection.from_pretrained(facebook/detr-resnet-50) model.to(cuda) optimizer = torch.optim.Adam(model.parameters()) x = X(optimizer)test()print(f{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()}) # Should print (, 0), but with cyclic reference, it will print (, ).``` 2022-09-27T15:28:52.2862986Z _=/usr/bin/env 2022-09-27T15:28:52.2863201Z + echo 'Testing pytorch' 2022-09-27T15:28:52.2863372Z Testing pytorch 2022-09-27T15:28:52.2863565Z + export LANG=C.UTF-8 2022-09-27T15:28:52.2863752Z + LANG=C.UTF-8 2022-09-27T15:28:52.2863940Z + PR_NUMBER=85462 2022-09-27T15:28:52.2864148Z + [[ distributed == \d\e\f\a\u\l\t ]] 2022-09-27T15:28:52.2864362Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2022-09-27T15:28:52.2864618Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-09-27T15:28:52.2864830Z + [[ distributed == \s\l\o\w ]] 2022-09-27T15:28:52.2865106Z + [[ linux-focal-py3.7-gcc7 == *slow-gradcheck* ]] 2022-09-27T15:28:52.2865375Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]] 2022-09-27T15:28:52.2865641Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-09-27T15:28:52.2865909Z + [[ linux-focal-py3.7-gcc7 == *cuda11* ]] 2022-09-27T15:28:52.2866162Z + [[ distributed == *crossref* ]] 2022-09-27T15:28:52.2866365Z + [[ distributed == *dynamo* ]] 2022-09-27T15:28:52.2866576Z + [[ -n 85462 ]] 2022-09-27T15:28:52.2866747Z + [[ -z '' ]] 2022-09-27T15:28:52.2866951Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1 2022-09-27T15:28:52.2867183Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1 2022-09-27T15:28:52.2867435Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-09-27T15:28:52.2867706Z + [[ linux-focal-py3.7-gcc7 != *-bazel-* ]] 2022-09-27T15:28:52.2867949Z + pip_install --user ninja 2022-09-27T15:28:52.2868206Z + pip install --progress-bar off --user ninja 2022-09-27T15:28:52.7017509Z Collecting ninja 2022-09-27T15:28:52.7177423Z Downloading ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2022-09-27T15:28:53.4395140Z Installing collected packages: ninja 2022-09-27T15:28:53.4482686Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2022-09-27T15:28:53.4483284Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-09-27T15:28:53.4530505Z Successfully installed ninja-1.10.2.3 2022-09-27T15:28:53.5256666Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-09-27T15:28:53.5257496Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-09-27T15:28:53.5258019Z + [[ linux-focal-py3.7-gcc7 == *asan* ]] 2022-09-27T15:28:53.5258265Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2022-09-27T15:28:53.5258502Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2022-09-27T15:28:53.5261209Z + [[ linux-focal-py3.7-gcc7 == *tbb* ]] 2022-09-27T15:28:53.5272460Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]] 2022-09-27T15:28:53.5273271Z + [[ linux-focal-py3.7-gcc7 == *-bazel-* ]] 2022-09-27T15:28:53.5275664Z + cd test 2022-09-27T15:28:53.5276153Z + python -c 'import torch; print(torch.__config__.show())' 2022-09-27T15:28:54.5686240Z PyTorch built with: 2022-09-27T15:28:54.5686693Z - GCC 7.5 2022-09-27T15:28:54.5686915Z - C++ Version: 201402 2022-09-27T15:28:54.5687341Z - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-09-27T15:28:54.5687758Z - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815) 2022-09-27T15:28:54.5688106Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-09-27T15:28:54.5688378Z - LAPACK is enabled (usually provided by MKL) 2022-09-27T15:28:54.5688603Z - NNPACK is enabled 2022-09-27T15:28:54.5688827Z - CPU capability usage: AVX2 2022-09-27T15:28:54.5691143Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 2022-09-27T15:28:54.5692830Z 2022-09-27T15:28:54.7433684Z + cd test 2022-09-27T15:28:54.7434149Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2022-09-27T15:28:55.7559046Z ATen/Parallel: 2022-09-27T15:28:55.7559644Z at::get_num_threads() : 4 2022-09-27T15:28:55.7559991Z at::get_num_interop_threads() : 4 2022-09-27T15:28:55.7560219Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-09-27T15:28:55.7560425Z omp_get_max_threads() : 4 2022-09-27T15:28:55.7560931Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-09-27T15:28:55.7561220Z mkl_get_max_threads() : 4 2022-09-27T15:28:55.7561544Z Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815) 2022-09-27T15:28:55.7561817Z std::thread::hardware_concurrency() : 8 2022-09-27T15:28:55.7562028Z Environment variables: 2022-09-27T15:28:55.7562213Z OMP_NUM_THREADS : [not set] 2022-09-27T15:28:55.7562411Z MKL_NUM_THREADS : [not set] 2022-09-27T15:28:55.7562622Z ATen parallel backend: OpenMP 2022-09-27T15:28:55.7562749Z 2022-09-27T15:28:55.9311497Z + [[ distributed == *deploy* ]] 2022-09-27T15:28:55.9312130Z + [[ distributed == *backward* ]] 2022-09-27T15:28:55.9312492Z + [[ distributed == *xla* ]] 2022-09-27T15:28:55.9312739Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2022-09-27T15:28:55.9313103Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]] 2022-09-27T15:28:55.9313337Z + [[ distributed == distributed ]] 2022-09-27T15:28:55.9313535Z + install_torchdynamo 2022-09-27T15:28:55.9313702Z + local commit 2022-09-27T15:28:55.9315782Z ++ get_pinned_commit torchdynamo 2022-09-27T15:28:55.9316214Z ++ cat .github/ci_commit_pins/torchdynamo.txt 2022-09-27T15:28:55.9327200Z + commit=41c44bc1d080d6cf063419a4166732b983b84eef 2022-09-27T15:28:55.9328079Z + pip_install --user git+https://github.com/pytorch/torchdynamo.git@41c44bc1d080d6cf063419a4166732b983b84eef 2022-09-27T15:28:55.9328855Z + pip install --progress-bar off --user git+https://github.com/pytorch/torchdynamo.git@41c44bc1d080d6cf063419a4166732b983b84eef 2022-09-27T15:28:56.2829036Z Collecting git+https://github.com/pytorch/torchdynamo.git@41c44bc1d080d6cf063419a4166732b983b84eef 2022-09-27T15:28:56.2835276Z Cloning https://github.com/pytorch/torchdynamo.git (to revision 41c44bc1d080d6cf063419a4166732b983b84eef) to /tmp/pip-req-build-cyou_aj7 2022-09-27T15:28:56.2867054Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchdynamo.git /tmp/pip-req-build-cyou_aj7 2022-09-27T15:28:56.9401836Z Running command git rev-parse -q --verify 'sha^41c44bc1d080d6cf063419a4166732b983b84eef' 2022-09-27T15:28:56.9434969Z Running command git fetch -q https://github.com/pytorch/torchdynamo.git 41c44bc1d080d6cf063419a4166732b983b84eef 2022-09-27T15:28:57.1998905Z Running command git checkout -q 41c44bc1d080d6cf063419a4166732b983b84eef 2022-09-27T15:28:57.4611205Z Resolved https://github.com/pytorch/torchdynamo.git to commit 41c44bc1d080d6cf063419a4166732b983b84eef 2022-09-27T15:28:59.2377350Z Preparing metadata (setup.py) ... [?25l- done 2022-09-27T15:28:59.2449504Z [?25hRequirement already satisfied: torch>=1.12.0 in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.13.0a0+git52424e2) 2022-09-27T15:28:59.2453796Z Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.18.5) 2022-09-27T15:28:59.2794855Z Collecting tabulate 2022-09-27T15:28:59.3064697Z Downloading tabulate-0.8.10-py3-none-any.whl (29 kB) 2022-09-27T15:28:59.3112716Z Requirement already satisfied: pyyaml in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (6.0) 2022-09-27T15:28:59.3117031Z Requirement already satisfied: sympy in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.10.1) 2022-09-27T15:28:59.3138727Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch>=1.12.0->torchdynamo==1.13.0.dev0) (4.3.0) 2022-09-27T15:28:59.3162961Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy->torchdynamo==1.13.0.dev0) (1.2.1) 2022-09-27T15:28:59.3260937Z Building wheels for collected packages: torchdynamo 2022-09-27T15:29:02.9363562Z Building wheel for torchdynamo (setup.py) ... [?25l- \ | / - done 2022-09-27T15:29:02.9440800Z [?25h Created wheel for torchdynamo: filename=torchdynamo-1.13.0.dev0-cp37-cp37m-linux_x86_64.whl size=2593921 sha256=949d57b28e6d3ade644d40b87465a4589735b19e8df61473e24f317f35942a2f 2022-09-27T15:29:02.9442436Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/77/0f/de/99455111d0619b9c9ed3eeb8e83431c83a6c69394197cf5fb8 2022-09-27T15:29:02.9469343Z Successfully built torchdynamo 2022-09-27T15:29:03.6449095Z Installing collected packages: tabulate, torchdynamo 2022-09-27T15:29:03.9012665Z Successfully installed tabulate-0.8.10 torchdynamo-1.13.0.dev0 2022-09-27T15:29:03.9861950Z + test_distributed 2022-09-27T15:29:03.9862586Z + echo 'Testing distributed python tests' 2022-09-27T15:29:03.9862925Z Testing distributed python tests 2022-09-27T15:29:03.9863275Z + python test/run_test.py --distributed-tests --shard 1 2 --verbose 2022-09-27T15:29:05.4204072Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda' 2022-09-27T15:29:05.4800679Z Ignoring disabled issues: [] 2022-09-27T15:29:05.4916937Z Found test time stats from artifacts 2022-09-27T15:29:05.4917366Z Selected tests: 2022-09-27T15:29:05.4917605Z distributed/rpc/test_tensorpipe_agent 2022-09-27T15:29:05.4917965Z distributed/rpc/cuda/test_tensorpipe_agent 2022-09-27T15:29:05.4918360Z distributed/fsdp/test_fsdp_state_dict 2022-09-27T15:29:05.4918766Z distributed/fsdp/test_fsdp_optim_state 2022-09-27T15:29:05.4919112Z distributed/test_c10d_nccl 2022-09-27T15:29:05.4919452Z distributed/fsdp/test_wrap 2022-09-27T15:29:05.4919838Z distributed/fsdp/test_fsdp_clip_grad_norm 2022-09-27T15:29:05.4920205Z distributed/fsdp/test_fsdp_misc 2022-09-27T15:29:05.4920668Z distributed/elastic/multiprocessing/api_test 2022-09-27T15:29:05.4921284Z distributed/test_store 2022-09-27T15:29:05.4921655Z distributed/algorithms/test_join 2022-09-27T15:29:05.4922021Z distributed/fsdp/test_fsdp_exec_order 2022-09-27T15:29:05.4922347Z distributed/fsdp/test_fsdp_freezing_weights 2022-09-27T15:29:05.4922568Z distributed/test_c10d_common 2022-09-27T15:29:05.4922784Z distributed/elastic/timer/local_timer_example 2022-09-27T15:29:05.4923053Z distributed/_shard/checkpoint/test_file_system_checkpoint_cpu 2022-09-27T15:29:05.4923325Z distributed/fsdp/test_fsdp_param_exec_order_wrap 2022-09-27T15:29:05.4923573Z distributed/_shard/sharded_tensor/test_sharded_tensor 2022-09-27T15:29:05.4923821Z distributed/elastic/timer/local_timer_test 2022-09-27T15:29:05.4924064Z distributed/fsdp/test_distributed_checkpoint 2022-09-27T15:29:05.4924307Z distributed/elastic/utils/distributed_test 2022-09-27T15:29:05.4924526Z distributed/fsdp/test_fsdp_pure_fp16 2022-09-27T15:29:05.4924742Z distributed/fsdp/test_fsdp_fx 2022-09-27T15:29:05.4924956Z distributed/fsdp/test_fsdp_uneven 2022-09-27T15:29:05.4925188Z distributed/_shard/sharding_spec/test_sharding_spec 2022-09-27T15:29:05.4925427Z distributed/fsdp/test_fsdp_overlap 2022-09-27T15:29:05.4925661Z distributed/_shard/checkpoint/test_planner 2022-09-27T15:29:05.4925885Z distributed/_shard/checkpoint/test_utils 2022-09-27T15:29:05.4926097Z distributed/test_launcher 2022-09-27T15:29:05.4926338Z distributed/_shard/sharded_tensor/ops/test_elementwise_ops 2022-09-27T15:29:05.4926588Z distributed/_shard/sharded_tensor/ops/test_init 2022-09-27T15:29:05.4926816Z distributed/test_c10d_spawn_nccl 2022-09-27T15:29:05.4927051Z distributed/_shard/sharded_tensor/ops/test_embedding 2022-09-27T15:29:05.4927310Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2022-09-27T15:29:05.4927583Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2022-09-27T15:29:05.4927845Z distributed/_shard/sharded_tensor/ops/test_softmax 2022-09-27T15:29:05.4928098Z distributed/_shard/sharded_tensor/ops/test_math_ops 2022-09-27T15:29:05.4928317Z distributed/_shard/test_sharder 2022-09-27T15:29:05.4928533Z distributed/elastic/timer/api_test 2022-09-27T15:29:05.4928766Z distributed/pipeline/sync/skip/test_api 2022-09-27T15:29:05.4929010Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2022-09-27T15:29:05.4929265Z distributed/pipeline/sync/skip/test_portal 2022-09-27T15:29:05.4929509Z distributed/pipeline/sync/skip/test_tracker 2022-09-27T15:29:05.4929732Z distributed/pipeline/sync/test_balance 2022-09-27T15:29:05.4929967Z distributed/pipeline/sync/test_checkpoint 2022-09-27T15:29:05.4930220Z distributed/pipeline/sync/test_deferred_batch_norm 2022-09-27T15:29:05.4930447Z distributed/pipeline/sync/test_inplace 2022-09-27T15:29:05.4930670Z distributed/pipeline/sync/test_phony 2022-09-27T15:29:05.4930901Z distributed/pipeline/sync/test_pipeline 2022-09-27T15:29:05.4931129Z distributed/pipeline/sync/test_transparency 2022-09-27T15:29:05.4931344Z distributed/test_nccl 2022-09-27T15:29:05.5062582Z Prioritized test from test file changes. 2022-09-27T15:29:05.5062927Z reordering tests for PR: 2022-09-27T15:29:05.5063338Z prioritized: ['distributed/fsdp/test_fsdp_optim_state'] 2022-09-27T15:29:05.5067158Z the rest: ['distributed/rpc/test_tensorpipe_agent', 'distributed/rpc/cuda/test_tensorpipe_agent', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/test_c10d_nccl', 'distributed/fsdp/test_wrap', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/fsdp/test_fsdp_misc', 'distributed/elastic/multiprocessing/api_test', 'distributed/test_store', 'distributed/algorithms/test_join', 'distributed/fsdp/test_fsdp_exec_order', 'distributed/fsdp/test_fsdp_freezing_weights', 'distributed/test_c10d_common', 'distributed/elastic/timer/local_timer_example', 'distributed/_shard/checkpoint/test_file_system_checkpoint_cpu', 'distributed/fsdp/test_fsdp_param_exec_order_wrap', 'distributed/_shard/sharded_tensor/test_sharded_tensor', 'distributed/elastic/timer/local_timer_test', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/elastic/utils/distributed_test', 'distributed/fsdp/test_fsdp_pure_fp16', 'distributed/fsdp/test_fsdp_fx', 'distributed/fsdp/test_fsdp_uneven', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/fsdp/test_fsdp_overlap', 'distributed/_shard/checkpoint/test_planner', 'distributed/_shard/checkpoint/test_utils', 'distributed/test_launcher', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops', 'distributed/_shard/sharded_tensor/ops/test_init', 'distributed/test_c10d_spawn_nccl', 'distributed/_shard/sharded_tensor/ops/test_embedding', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag', 'distributed/_shard/sharded_tensor/ops/test_softmax', 'distributed/_shard/sharded_tensor/ops/test_math_ops', 'distributed/_shard/test_sharder', 'distributed/elastic/timer/api_test', 'distributed/pipeline/sync/skip/test_api', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_transparency', 'distributed/test_nccl'] 2022-09-27T15:29:05.5069628Z 2022-09-27T15:29:05.5070044Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2022-09-27T15:29:05.5435334Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2022-09-27T15:29:05.5584319Z Running distributed/fsdp/test_fsdp_optim_state ... [2022-09-27 15:29:05.558197] 2022-09-27T15:29:05.5584871Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:29:05.558241] 2022-09-27T15:29:06.7880882Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state 2022-09-27T15:29:06.7897007Z 2022-09-27T15:29:06.7897425Z Running tests... 2022-09-27T15:29:06.7897823Z ---------------------------------------------------------------------- 2022-09-27T15:29:06.7903970Z test_flatten_sharded_optim_state_dict_nested (__main__.TestFSDPOptimState) 2022-09-27T15:29:07.0340892Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 693 2022-09-27T15:29:07.0365635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 694 2022-09-27T15:29:07.0391062Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 695 2022-09-27T15:29:07.0417148Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 696 2022-09-27T15:29:08.2833742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:08.2834396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:08.2836653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:08.2837229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:08.2935112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:08.2935770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:08.2939990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:08.2940604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:08.3487308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:08.3487948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:08.3490872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:08.3491478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:08.3509121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:08.3509764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:08.3513990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:08.3514858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:08.4132276Z dist init r=1, world=4 2022-09-27T15:29:08.4227180Z dist init r=3, world=4 2022-09-27T15:29:08.4771967Z dist init r=2, world=4 2022-09-27T15:29:08.4785953Z dist init r=0, world=4 2022-09-27T15:29:08.5082569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:08.5185985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:08.5186913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:08.5187385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:08.5187753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:08.5188264Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:08.5188875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:08.5189410Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:08.5191212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:08.5191792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:08.5192325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:08.5192857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:08.8464272Z skip: Need at least 2 CUDA devices (2.056s) 2022-09-27T15:29:08.8469802Z test_flatten_sharded_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-09-27T15:29:08.8506309Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 749 2022-09-27T15:29:08.8532977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 750 2022-09-27T15:29:08.8560136Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 751 2022-09-27T15:29:08.8593615Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 752 2022-09-27T15:29:10.0134045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:10.0134764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:10.0137582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:10.0138278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:10.0241112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:10.0242044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:10.0246066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:10.0246867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:10.0340060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:10.0340661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:10.0344365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:10.0344979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:10.0432274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:10.0432925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:10.0437550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:10.0438183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:10.1419734Z dist init r=2, world=4 2022-09-27T15:29:10.1537505Z dist init r=1, world=4 2022-09-27T15:29:10.1617224Z dist init r=3, world=4 2022-09-27T15:29:10.1698563Z dist init r=0, world=4 2022-09-27T15:29:10.1929987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:10.2031384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:10.2133297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:10.2134198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:10.2134843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:10.2135615Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:10.2136314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:10.2136938Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:10.2140845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:10.2142014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:10.2142539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:10.2143057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:10.5635403Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:10.5646404Z test_full_optim_state_dict_keys (__main__.TestFSDPOptimState) 2022-09-27T15:29:10.5683166Z Tests that the parameter keys returned by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 805 2022-09-27T15:29:10.5709629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 806 2022-09-27T15:29:10.5735654Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 807 2022-09-27T15:29:10.5762471Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 808 2022-09-27T15:29:11.7421979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:11.7422599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:11.7425725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:11.7426353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:11.7560825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:11.7561692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:11.7565217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:11.7566181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:11.7992154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:11.7992734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:11.7995555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:11.7996180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:11.8203074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:11.8203743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:11.8207660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:11.8208325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:11.8707428Z dist init r=3, world=4 2022-09-27T15:29:11.8848182Z dist init r=0, world=4 2022-09-27T15:29:11.9437807Z dist init r=1, world=4 2022-09-27T15:29:11.9513926Z dist init r=2, world=4 2022-09-27T15:29:11.9828973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:11.9829861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:11.9847244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:11.9848299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:11.9848776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:11.9849271Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:11.9849779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:11.9850297Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:11.9850686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:11.9851034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:11.9851368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:11.9855292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:12.2804154Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:12.2812127Z test_full_optim_state_dict_nested_invalid (__main__.TestFSDPOptimState) 2022-09-27T15:29:12.2848648Z Tests that :meth:`full_optim_state_dict` raises an error when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 861 2022-09-27T15:29:12.2877485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 862 2022-09-27T15:29:12.2903963Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 863 2022-09-27T15:29:12.2932432Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 864 2022-09-27T15:29:13.4183066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:13.4183621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:13.4185811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:13.4186323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:13.4245647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:13.4246276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:13.4250355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:13.4250986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:13.4503232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:13.4503858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:13.4507457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:13.4508098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:13.4538420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:13.4539023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:13.4543277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:13.5472646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:13.5473126Z dist init r=1, world=4 2022-09-27T15:29:13.5535806Z dist init r=3, world=4 2022-09-27T15:29:13.5800431Z dist init r=0, world=4 2022-09-27T15:29:13.5817502Z dist init r=2, world=4 2022-09-27T15:29:13.6047510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:13.6148832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:13.6149523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:13.6150200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:13.6151239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:13.6152115Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:13.6152644Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:13.6153166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:13.6256677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:13.6257256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:13.6257803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:13.6258228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:13.9972483Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:13.9986014Z test_optim_input_warning (__main__.TestFSDPOptimState) 2022-09-27T15:29:14.0021509Z Tests that passing the ``optim_input`` argument into optimizer state ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 917 2022-09-27T15:29:14.0048775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 918 2022-09-27T15:29:14.0074949Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 919 2022-09-27T15:29:14.0102587Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 920 2022-09-27T15:29:15.1377456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:15.1378150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:15.1382094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:15.1382710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:15.1408673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:15.1409195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:15.1414886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:15.1415419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:15.1731342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:15.1732016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:15.1736391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:15.1737023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:15.2004921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:15.2005598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:15.2009496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:15.2010130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:15.2694721Z dist init r=2, world=4 2022-09-27T15:29:15.2695126Z dist init r=1, world=4 2022-09-27T15:29:15.3036000Z dist init r=3, world=4 2022-09-27T15:29:15.3294168Z dist init r=0, world=4 2022-09-27T15:29:15.3609907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:15.3710421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:15.3813456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:15.3814366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:15.3814888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:15.3815455Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:15.3816043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:15.3816600Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:15.3823504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:15.3824885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:15.3825393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:15.3825729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:15.7143525Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:15.7152060Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:29:15.7188997Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 973 2022-09-27T15:29:15.7217323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 974 2022-09-27T15:29:15.7250557Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 975 2022-09-27T15:29:15.7281698Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 976 2022-09-27T15:29:16.8905671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:16.8906353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:16.8910267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:16.8910908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:16.8960835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:16.8961498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:16.8965150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:16.8965824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:16.9243628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:16.9244263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:16.9247226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:16.9247859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:16.9510297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:16.9510946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:16.9514822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:16.9515463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:17.0228544Z dist init r=0, world=4 2022-09-27T15:29:17.0247542Z dist init r=3, world=4 2022-09-27T15:29:17.0540452Z dist init r=2, world=4 2022-09-27T15:29:17.0789390Z dist init r=1, world=4 2022-09-27T15:29:17.1054148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:17.1155475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:17.1156073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:17.1156627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:17.1157382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:17.1157917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:17.1158519Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:17.1159039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:17.1164325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:17.1164918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:17.1165716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:17.1166525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:17.4323460Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T15:29:17.4332268Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:29:17.4368801Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1029 2022-09-27T15:29:17.4395541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1030 2022-09-27T15:29:17.4422003Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1031 2022-09-27T15:29:17.4449548Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1032 2022-09-27T15:29:18.5873759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:18.5874327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:18.5877528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:18.5878159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:18.5996838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:18.5997475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:18.6001356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:18.6001984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:18.6187962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:18.6188566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:18.6192344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:18.6193018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:18.6286655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:18.6287306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:18.6291600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:18.6292257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:18.7178881Z dist init r=0, world=4 2022-09-27T15:29:18.7296398Z dist init r=1, world=4 2022-09-27T15:29:18.7474157Z dist init r=2, world=4 2022-09-27T15:29:18.7595236Z dist init r=3, world=4 2022-09-27T15:29:18.7705675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:18.7809259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:18.7911781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:18.7912475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:18.7913077Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:18.7913764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:18.7914290Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:18.8009696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:18.8018557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:18.8019714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:18.8020573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:18.8020978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:19.1491544Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:19.1499962Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:29:19.1535409Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1085 2022-09-27T15:29:19.1562023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1086 2022-09-27T15:29:19.1588947Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1087 2022-09-27T15:29:19.1622220Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1088 2022-09-27T15:29:20.2712082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:20.2712741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:20.2716203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:20.2716838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:20.2885015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:20.2885671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:20.2889122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:20.2889733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:20.3233798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:20.3234426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:20.3237921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:20.3238584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:20.3359568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:20.3360162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:20.3363284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:20.3363889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:20.4007887Z dist init r=0, world=4 2022-09-27T15:29:20.4161279Z dist init r=3, world=4 2022-09-27T15:29:20.4503129Z dist init r=1, world=4 2022-09-27T15:29:20.4623385Z dist init r=2, world=4 2022-09-27T15:29:20.4833047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:20.4934231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:20.5036339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:20.5037484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:20.5038249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:20.5038887Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:20.5039614Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:20.5040126Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:20.5143438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:20.5144100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:20.5144594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:20.5145005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:20.8663874Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:20.8672133Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:29:20.8710767Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1141 2022-09-27T15:29:20.8737685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1142 2022-09-27T15:29:20.8764798Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1143 2022-09-27T15:29:20.8792204Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1144 2022-09-27T15:29:21.9898867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:21.9899553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:21.9903190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:21.9903798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:22.0473724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:22.0474342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:22.0477499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:22.0478139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:22.0563180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:22.0563797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:22.0567658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:22.0568275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:22.0585660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:22.0586260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:22.0590961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:22.0591665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:22.1189069Z dist init r=2, world=4 2022-09-27T15:29:22.1764303Z dist init r=0, world=4 2022-09-27T15:29:22.1848154Z dist init r=1, world=4 2022-09-27T15:29:22.1871520Z dist init r=3, world=4 2022-09-27T15:29:22.1981655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:22.2083002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:22.2184729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:22.2186952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:22.2188262Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:22.2188805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:22.2189328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:22.2189858Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:22.2300040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:22.2300648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:22.2301161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:22.2314315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:22.5833636Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:22.5842126Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:29:22.5880943Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1197 2022-09-27T15:29:22.5911705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1198 2022-09-27T15:29:22.5939184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1199 2022-09-27T15:29:22.5972488Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1200 2022-09-27T15:29:23.8242576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:23.8243217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:23.8246612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:23.8247224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:23.8286680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:23.8287340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:23.8291988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:23.8292618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:23.8320310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:23.8321331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:23.8325109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:23.8325730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:23.8616064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:23.8616668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:23.8619010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:23.8619582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:23.9527241Z dist init r=1, world=4 2022-09-27T15:29:23.9572256Z dist init r=0, world=4 2022-09-27T15:29:23.9589115Z dist init r=2, world=4 2022-09-27T15:29:23.9895916Z dist init r=3, world=4 2022-09-27T15:29:24.0039453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:24.0039861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:24.0143191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:24.0143891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:24.0144973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:24.0145787Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:24.0146296Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:24.0146813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:24.0150105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:24.0150736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:24.0151260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:24.0151841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:24.3014270Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T15:29:24.3022064Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:29:24.3056813Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1253 2022-09-27T15:29:24.3083320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1254 2022-09-27T15:29:24.3109917Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1255 2022-09-27T15:29:24.3136159Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1256 2022-09-27T15:29:25.5361020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:25.5361672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:25.5364751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:25.5365419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:25.5633870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:25.5634460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:25.5637398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:25.5637936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:25.5670525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:25.5671181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:25.5676328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:25.5676949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:25.6093160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:25.6093759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:25.6097539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:25.6098183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:25.6644077Z dist init r=0, world=4 2022-09-27T15:29:25.6920721Z dist init r=1, world=4 2022-09-27T15:29:25.6972400Z dist init r=2, world=4 2022-09-27T15:29:25.7364231Z dist init r=3, world=4 2022-09-27T15:29:25.7472566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:25.7534051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:25.7635508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:25.7636044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:25.7636665Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:25.7637183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:25.7637708Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:25.7675541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:25.7743410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:25.7743947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:25.7744492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:25.7744937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:26.1179195Z skip: Need at least 2 CUDA devices (1.816s) 2022-09-27T15:29:26.1187665Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:29:26.1225528Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1309 2022-09-27T15:29:26.1252743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1310 2022-09-27T15:29:26.1280030Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1311 2022-09-27T15:29:26.1308059Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1312 2022-09-27T15:29:27.2819077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:27.2819605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:27.2822658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:27.2823170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:27.2947519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:27.2948179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:27.2948964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:27.2949595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:27.2951298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:27.2951991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:27.2952792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:27.2953351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:27.3219455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:27.3220098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:27.3223740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:27.3224399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:27.4100872Z dist init r=2, world=4 2022-09-27T15:29:27.4229056Z dist init r=0, world=4 2022-09-27T15:29:27.4397487Z dist init r=1, world=4 2022-09-27T15:29:27.4498422Z dist init r=3, world=4 2022-09-27T15:29:27.4707775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:27.4809059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:27.4809538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:27.4810165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:27.4810851Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:27.4811446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:27.4811960Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:27.4812479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:27.4917488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:27.4917933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:27.4918286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:27.4918818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:27.8349237Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:27.8357970Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:29:27.8395043Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1365 2022-09-27T15:29:27.8422426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1366 2022-09-27T15:29:27.8448774Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1367 2022-09-27T15:29:27.8476415Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1368 2022-09-27T15:29:29.0015189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:29.0015805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:29.0018300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:29.0018805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:29.0263269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:29.0263946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:29.0267653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:29.0268011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:29.0353459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:29.0353815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:29.0360300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:29.0360950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:29.0413205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:29.0413823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:29.0417757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:29.0418373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:29.1304376Z dist init r=2, world=4 2022-09-27T15:29:29.1712773Z dist init r=0, world=4 2022-09-27T15:29:29.1845737Z dist init r=1, world=4 2022-09-27T15:29:29.1910368Z dist init r=3, world=4 2022-09-27T15:29:29.2226236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:29.2328109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:29.2328850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:29.2329508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:29.2330286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:29.2330800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:29.2331317Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:29.2331841Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:29.2435142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:29.2435692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:29.2436222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:29.2436774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:29.5517019Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:29.5525912Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:29:29.5573071Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1421 2022-09-27T15:29:29.5600253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1422 2022-09-27T15:29:29.5627798Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1423 2022-09-27T15:29:29.5662009Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1424 2022-09-27T15:29:30.6612786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:30.6613441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:30.6617340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:30.6617984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:30.7108883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:30.7109544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:30.7112670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:30.7113424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:30.7225777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:30.7226406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:30.7230721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:30.7231363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:30.7337400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:30.7338028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:30.7342721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:30.7343365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:30.7900604Z dist init r=0, world=4 2022-09-27T15:29:30.8385174Z dist init r=1, world=4 2022-09-27T15:29:30.8506508Z dist init r=3, world=4 2022-09-27T15:29:30.8603306Z dist init r=2, world=4 2022-09-27T15:29:30.8715499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:30.8816441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:30.8918659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:30.8919235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:30.8919852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:30.8920578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:30.8921497Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:30.8922069Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:30.9026427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:30.9026973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:30.9027503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:30.9028031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:31.2703428Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T15:29:31.2711359Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:29:31.2747437Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1477 2022-09-27T15:29:31.2774602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1478 2022-09-27T15:29:31.2801341Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1479 2022-09-27T15:29:31.2828918Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1480 2022-09-27T15:29:32.4168110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:32.4168708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:32.4172241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:32.4172865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:32.4475003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:32.4475653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:32.4478429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:32.4479056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:32.4580531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:32.4581200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:32.4584843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:32.4585460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:32.4589332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:32.4589915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:32.4594739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:32.4595379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:32.5447566Z dist init r=3, world=4 2022-09-27T15:29:32.5763792Z dist init r=2, world=4 2022-09-27T15:29:32.5863857Z dist init r=1, world=4 2022-09-27T15:29:32.5875594Z dist init r=0, world=4 2022-09-27T15:29:32.6175148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:32.6276757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:32.6277460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:32.6277980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:32.6278791Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:32.6279488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:32.6280059Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:32.6280590Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:32.6383712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:32.6384283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:32.6384951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:32.6385473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:32.9869190Z skip: Need at least 2 CUDA devices (1.716s) 2022-09-27T15:29:32.9877784Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:29:32.9914466Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1533 2022-09-27T15:29:32.9942003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1534 2022-09-27T15:29:32.9969110Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1535 2022-09-27T15:29:32.9997292Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1536 2022-09-27T15:29:34.1158706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:34.1159382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:34.1162500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:34.1163178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:34.1441076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:34.1441714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:34.1444812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:34.1445446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:34.1646380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:34.1647027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:34.1650823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:34.1651452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:34.1700877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:34.1701481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:34.1706104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:34.1706771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:34.2443051Z dist init r=3, world=4 2022-09-27T15:29:34.2715761Z dist init r=0, world=4 2022-09-27T15:29:34.2924327Z dist init r=2, world=4 2022-09-27T15:29:34.2959470Z dist init r=1, world=4 2022-09-27T15:29:34.3234273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:34.3327480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:34.3429712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:34.3430641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:34.3431053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:34.3431800Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:34.3432320Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:34.3436923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:34.3537410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:34.3538001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:34.3538529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:34.3539068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:34.7037806Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:34.7046248Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:29:34.7080318Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1589 2022-09-27T15:29:34.7107290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1590 2022-09-27T15:29:34.7134385Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1591 2022-09-27T15:29:34.7168094Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1592 2022-09-27T15:29:35.8473899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:35.8474670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:35.8477587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:35.8478127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:35.8612862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:35.8613514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:35.8617004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:35.8617600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:35.8735435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:35.8736071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:35.8739807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:35.8740447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:35.8871930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:35.8872610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:35.8876952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:35.8877574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:35.9761047Z dist init r=2, world=4 2022-09-27T15:29:35.9894281Z dist init r=3, world=4 2022-09-27T15:29:36.0013386Z dist init r=1, world=4 2022-09-27T15:29:36.0141645Z dist init r=0, world=4 2022-09-27T15:29:36.0373208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:36.0474444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:36.0475415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:36.0475804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:36.0476682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:36.0477220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:36.0477731Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:36.0478257Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:36.0581437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:36.0582319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:36.0582998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:36.0583550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:36.4207599Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:36.4216969Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:29:36.4254075Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1645 2022-09-27T15:29:36.4281189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1646 2022-09-27T15:29:36.4307458Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1647 2022-09-27T15:29:36.4335381Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1648 2022-09-27T15:29:37.6008568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:37.6009231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:37.6012007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:37.6012663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:37.6728801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:37.6729442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:37.6731965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:37.6732631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:37.6749491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:37.6750087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:37.6754449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:37.6755331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:37.6807318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:37.6807889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:37.6811763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:37.6812364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:37.7322051Z dist init r=3, world=4 2022-09-27T15:29:37.8010905Z dist init r=1, world=4 2022-09-27T15:29:37.8038437Z dist init r=2, world=4 2022-09-27T15:29:37.8081766Z dist init r=0, world=4 2022-09-27T15:29:37.8349182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:37.8452053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:37.8452899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:37.8453469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:37.8454290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:37.8454857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:37.8455585Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:37.8456386Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:37.8459724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:37.8460347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:37.8460846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:37.8461323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:38.1376974Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:38.1384438Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:29:38.1420818Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1701 2022-09-27T15:29:38.1448700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1702 2022-09-27T15:29:38.1475945Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1703 2022-09-27T15:29:38.1503767Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1704 2022-09-27T15:29:39.2470466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:39.2471120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:39.2474897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:39.2475501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:39.2985997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:39.2986602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:39.2989759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:39.2990362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:39.3152077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:39.3152704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:39.3156017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:39.3156654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:39.3185142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:39.3185708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:39.3189623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:39.3190250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:39.3755161Z dist init r=1, world=4 2022-09-27T15:29:39.4267069Z dist init r=2, world=4 2022-09-27T15:29:39.4424063Z dist init r=3, world=4 2022-09-27T15:29:39.4455282Z dist init r=0, world=4 2022-09-27T15:29:39.4677264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:39.4780175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:39.4780855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:39.4781794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:39.4782357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:39.4783065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:39.4783587Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:39.4784106Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:39.4788415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:39.4788980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:39.4789523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:39.4790066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:39.7543444Z skip: Need at least 2 CUDA devices (1.617s) 2022-09-27T15:29:39.7551136Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:29:39.7587578Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1757 2022-09-27T15:29:39.7614313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1758 2022-09-27T15:29:39.7640799Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1759 2022-09-27T15:29:39.7668088Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1760 2022-09-27T15:29:40.9178117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:40.9178792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:40.9182619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:40.9183257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:40.9859112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:40.9859768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:40.9863619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:40.9864255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:40.9934990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:40.9935684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:40.9940168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:40.9940803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:40.9967254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:40.9967880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:40.9971453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:40.9972074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:41.0473131Z dist init r=1, world=4 2022-09-27T15:29:41.1125464Z dist init r=0, world=4 2022-09-27T15:29:41.1198031Z dist init r=2, world=4 2022-09-27T15:29:41.1228769Z dist init r=3, world=4 2022-09-27T15:29:41.1408220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:41.1509497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:41.1610501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:41.1611302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:41.1612220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:41.1612826Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:41.1613360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:41.1613885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:41.1718523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:41.1719128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:41.1719679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:41.1720185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:41.4708926Z skip: Need at least 2 CUDA devices (1.716s) 2022-09-27T15:29:41.4717611Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:29:41.4753955Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1813 2022-09-27T15:29:41.4781168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1814 2022-09-27T15:29:41.4808328Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1815 2022-09-27T15:29:41.4836081Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1816 2022-09-27T15:29:42.5886791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:42.5887324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:42.5891547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:42.5892173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:42.6016829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:42.6017338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:42.6021403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:42.6022042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:42.6382283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:42.6382910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:42.6385819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:42.6386406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:42.6522393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:42.6523016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:42.6526307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:42.6526918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:42.7189601Z dist init r=3, world=4 2022-09-27T15:29:42.7306370Z dist init r=0, world=4 2022-09-27T15:29:42.7668043Z dist init r=1, world=4 2022-09-27T15:29:42.7791883Z dist init r=2, world=4 2022-09-27T15:29:42.7900708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:42.8103037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:42.8205400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:42.8205851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:42.8206656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:42.8207275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:42.8207798Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:42.8208325Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:42.8312544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:42.8313180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:42.8313574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:42.8314018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:43.1877597Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:43.1884513Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState) 2022-09-27T15:29:43.1921892Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1869 2022-09-27T15:29:43.1949345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1870 2022-09-27T15:29:43.1976394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1871 2022-09-27T15:29:43.2004915Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1872 2022-09-27T15:29:44.3383379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:44.3384300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:44.3387594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:44.3388393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:44.3576115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:44.3576719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:44.3580342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:44.3580996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:44.3651998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:44.3652648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:44.3656067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:44.3656709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:44.3658223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:44.3658814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:44.3663595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:44.3664249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:44.4683262Z dist init r=2, world=4 2022-09-27T15:29:44.4868418Z dist init r=1, world=4 2022-09-27T15:29:44.4933785Z dist init r=0, world=4 2022-09-27T15:29:44.4947265Z dist init r=3, world=4 2022-09-27T15:29:44.5092281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:44.5281319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:44.5281724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:44.5282095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:44.5282720Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:44.5283254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:44.5283761Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:44.5295927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:44.5388222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:44.5388877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:44.5389219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:44.5389553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:44.9044826Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:44.9051085Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState) 2022-09-27T15:29:44.9087709Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1925 2022-09-27T15:29:44.9115156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1926 2022-09-27T15:29:44.9142104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1927 2022-09-27T15:29:44.9170090Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1928 2022-09-27T15:29:46.0648897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:46.0649594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:46.0653017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:46.0653658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:46.1143008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:46.1143582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:46.1147614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:46.1148500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:46.1235458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:46.1236141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:46.1240728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:46.1241377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:46.1573207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:46.1573867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:46.1576503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:46.1577126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:46.1948955Z dist init r=1, world=4 2022-09-27T15:29:46.2468510Z dist init r=3, world=4 2022-09-27T15:29:46.2553198Z dist init r=0, world=4 2022-09-27T15:29:46.2850561Z dist init r=2, world=4 2022-09-27T15:29:46.3060381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:46.3065667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:46.3066822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:46.3067890Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:46.3068491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:46.3069006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:46.3069531Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:46.3162106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:46.3174080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:46.3174646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:46.3175165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:46.3196325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:46.6210682Z skip: Need at least 2 CUDA devices (1.716s) 2022-09-27T15:29:46.6217361Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState) 2022-09-27T15:29:46.6253389Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1981 2022-09-27T15:29:46.6280354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1982 2022-09-27T15:29:46.6307201Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1983 2022-09-27T15:29:46.6335179Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1984 2022-09-27T15:29:47.8076077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:47.8076699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:47.8080482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:47.8081094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:47.8118699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:47.8119293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:47.8123119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:47.8123744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:47.8207420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:47.8208013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:47.8212616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:47.8213238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:47.8214034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:47.8214593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:47.8218469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:47.8219098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:47.9374623Z dist init r=0, world=4 2022-09-27T15:29:47.9392900Z dist init r=3, world=4 2022-09-27T15:29:47.9484408Z dist init r=2, world=4 2022-09-27T15:29:47.9496288Z dist init r=1, world=4 2022-09-27T15:29:47.9796258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:47.9896682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:47.9999728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:48.0000385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:48.0001011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:48.0001526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:48.0002045Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:48.0002558Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:48.0107068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:48.0107924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:48.0108281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:48.0108694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:48.3376462Z skip: Need at least 2 CUDA devices (1.716s) 2022-09-27T15:29:48.3382713Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState) 2022-09-27T15:29:48.3419753Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2037 2022-09-27T15:29:48.3448258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2038 2022-09-27T15:29:48.3475311Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2039 2022-09-27T15:29:48.3503139Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2040 2022-09-27T15:29:49.4567256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:49.4567934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:49.4570989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:49.4571604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:49.4641829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:49.4642440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:49.4646014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:49.4646633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:49.4936234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:49.4936909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:49.4940905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:49.4941512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:49.5197126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:49.5197802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:49.5200369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:49.5201007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:49.5859766Z dist init r=2, world=4 2022-09-27T15:29:49.5930636Z dist init r=1, world=4 2022-09-27T15:29:49.6218419Z dist init r=0, world=4 2022-09-27T15:29:49.6470586Z dist init r=3, world=4 2022-09-27T15:29:49.6644433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:49.6644854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:49.6746560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:49.6747263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:49.6747908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:49.6748452Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:49.6749144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:49.6749665Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:49.6854926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:49.6855516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:49.6856046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:49.6856405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:50.0544744Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:50.0548904Z test_rekey_optim_state_dict_to_names (__main__.TestFSDPOptimState) 2022-09-27T15:29:50.0586585Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2093 2022-09-27T15:29:50.0614171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2094 2022-09-27T15:29:50.0640884Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2095 2022-09-27T15:29:50.0668439Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2096 2022-09-27T15:29:51.2333563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:51.2334292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:51.2337255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:51.2337941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:51.2394158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:51.2394945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:51.2399077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:51.2399709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:51.2557947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:51.2558564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:51.2561940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:51.2562549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:51.2723287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:51.2723949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:51.2727438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:51.2728080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:51.3634409Z dist init r=1, world=4 2022-09-27T15:29:51.3671890Z dist init r=0, world=4 2022-09-27T15:29:51.3991760Z dist init r=2, world=4 2022-09-27T15:29:51.4002532Z dist init r=3, world=4 2022-09-27T15:29:51.4247541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:51.4348634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:51.4452043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:51.4452730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:51.4453789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:51.4454628Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:51.4455410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:51.4456220Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:51.4459902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:51.4460525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:51.4461127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:51.4461646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:51.7709792Z skip: Need at least 2 CUDA devices (1.716s) 2022-09-27T15:29:51.7717077Z test_scatter_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState) 2022-09-27T15:29:51.7753430Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2149 2022-09-27T15:29:51.7780602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2150 2022-09-27T15:29:51.7807055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2151 2022-09-27T15:29:51.7834403Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2152 2022-09-27T15:29:52.9260647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:52.9261293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:52.9263576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:52.9264160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:52.9374284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:52.9374942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:52.9379456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:52.9380094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:52.9661032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:52.9661603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:52.9663797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:52.9664357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:52.9665107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:52.9665739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:52.9670094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:52.9670708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:53.0548147Z dist init r=2, world=4 2022-09-27T15:29:53.0645124Z dist init r=1, world=4 2022-09-27T15:29:53.0931651Z dist init r=3, world=4 2022-09-27T15:29:53.0935936Z dist init r=0, world=4 2022-09-27T15:29:53.1159338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:53.1261379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:53.1362663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:53.1363681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:53.1364092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:53.1364591Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:53.1365103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:53.1365623Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:53.1471435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:53.1472043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:53.1472437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:53.1472799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:53.4876339Z skip: Need at least 2 CUDA devices (1.716s) 2022-09-27T15:29:53.4883772Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:29:53.4920643Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2205 2022-09-27T15:29:53.4947510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2206 2022-09-27T15:29:53.4974568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2207 2022-09-27T15:29:53.5001405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2208 2022-09-27T15:29:54.7424567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:54.7425287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:54.7426146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:54.7426784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:54.7427637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:54.7428331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:54.7429201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:54.7429889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:54.7630173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:54.7630841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:54.7634943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:54.7635595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:54.7768776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:54.7769374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:54.7773287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:54.7774182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:54.8706946Z dist init r=1, world=4 2022-09-27T15:29:54.8707744Z dist init r=0, world=4 2022-09-27T15:29:54.8905073Z dist init r=3, world=4 2022-09-27T15:29:54.9033697Z dist init r=2, world=4 2022-09-27T15:29:54.9214771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:54.9315874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:54.9318197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:54.9318592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:54.9319207Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:54.9319755Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:54.9417947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:54.9418817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:54.9425368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:54.9426045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:54.9426468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:54.9426833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:55.3044400Z skip: Need at least 2 CUDA devices (1.817s) 2022-09-27T15:29:55.3052439Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:29:55.3090124Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2261 2022-09-27T15:29:55.3117299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2262 2022-09-27T15:29:55.3144645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2263 2022-09-27T15:29:55.3172835Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2264 2022-09-27T15:29:56.4804045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:56.4804666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:56.4807429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:56.4808047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:56.5011042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:56.5011656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:56.5015593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:56.5016221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:56.5094112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:56.5094707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:56.5099249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:56.5099840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:56.5274001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:56.5274804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:56.5278836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:56.5279447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:56.6130514Z dist init r=2, world=4 2022-09-27T15:29:56.6325873Z dist init r=1, world=4 2022-09-27T15:29:56.6390685Z dist init r=0, world=4 2022-09-27T15:29:56.6592103Z dist init r=3, world=4 2022-09-27T15:29:56.6738722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:56.6739120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:56.6840650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:56.6841823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:56.6842677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:56.6843283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:56.6843837Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:56.6844345Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:56.6847283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:56.6848101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:56.6850098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:56.6850899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:57.0215010Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:57.0222510Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:29:57.0261129Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2317 2022-09-27T15:29:57.0288783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2318 2022-09-27T15:29:57.0315700Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2319 2022-09-27T15:29:57.0343677Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2320 2022-09-27T15:29:58.1444230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:58.1444827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:58.1447530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:58.1448097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:58.1564199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:58.1564834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:58.1568112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:58.1568720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:58.1679598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:58.1680146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:58.1683104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:58.1683601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:58.2146309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:58.2146949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:58.2150816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:58.2151512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:58.2744412Z dist init r=2, world=4 2022-09-27T15:29:58.2857986Z dist init r=3, world=4 2022-09-27T15:29:58.2958883Z dist init r=0, world=4 2022-09-27T15:29:58.3416387Z dist init r=1, world=4 2022-09-27T15:29:58.3671821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:29:58.3772410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:29:58.3876773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:58.3877421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:29:58.3878212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:58.3878813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:29:58.3879383Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:58.3879918Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:29:58.3883182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:29:58.3883692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:29:58.3884186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:29:58.3884727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:29:58.7385230Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:29:58.7393300Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:29:58.7431840Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2373 2022-09-27T15:29:58.7466673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2374 2022-09-27T15:29:58.7499523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2375 2022-09-27T15:29:58.7531616Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2376 2022-09-27T15:29:59.8995652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:59.8996287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:59.8997012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:59.8997555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:59.8998293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:59.8999134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:59.9001086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:59.9001789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:59.9163659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:59.9164269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:59.9167298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:59.9167986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:29:59.9254349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:29:59.9254952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:29:59.9258808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:29:59.9259447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:00.0279191Z dist init r=2, world=4 2022-09-27T15:30:00.0279549Z dist init r=0, world=4 2022-09-27T15:30:00.0440950Z dist init r=3, world=4 2022-09-27T15:30:00.0528133Z dist init r=1, world=4 2022-09-27T15:30:00.0751906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:00.0852810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:00.0955937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:00.0956815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:00.0957449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:00.0958094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:00.0958731Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:00.0959517Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:00.0962691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:00.0963239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:00.0963601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:00.0963991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:00.4572835Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T15:30:00.4580790Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:30:00.4618511Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2429 2022-09-27T15:30:00.4646237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2430 2022-09-27T15:30:00.4673175Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2431 2022-09-27T15:30:00.4701087Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2432 2022-09-27T15:30:01.6243415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:01.6244144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:01.6247582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:01.6248438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:01.6442788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:01.6443436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:01.6446886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:01.6447515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:01.6467073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:01.6467704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:01.6471754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:01.6472368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:01.6654079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:01.6654707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:01.6658980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:01.6659647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:01.7745790Z dist init r=1, world=4 2022-09-27T15:30:01.7770046Z dist init r=3, world=4 2022-09-27T15:30:01.7826792Z dist init r=2, world=4 2022-09-27T15:30:01.8028729Z dist init r=0, world=4 2022-09-27T15:30:01.8181725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:01.8282790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:01.8384525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:01.8385053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:01.8385939Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:01.8386677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:01.8387268Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:01.8387792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:01.8494562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:01.8495147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:01.8495651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:01.8496247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:02.1742282Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:02.1750028Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:30:02.1754943Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85092 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-09-27T15:30:02.1760331Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:30:02.1796445Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2485 2022-09-27T15:30:02.1824666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2486 2022-09-27T15:30:02.1851830Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2487 2022-09-27T15:30:02.1880223Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2488 2022-09-27T15:30:03.2976004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:03.2976607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:03.2979833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:03.2980459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:03.3009093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:03.3010145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:03.3014654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:03.3015369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:03.3121448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:03.3122099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:03.3126742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:03.3127370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:03.3546414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:03.3547028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:03.3550640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:03.3551288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:03.4283698Z dist init r=3, world=4 2022-09-27T15:30:03.4297668Z dist init r=0, world=4 2022-09-27T15:30:03.4414221Z dist init r=2, world=4 2022-09-27T15:30:03.4834649Z dist init r=1, world=4 2022-09-27T15:30:03.5127146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:03.5229747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:03.5230513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:03.5241041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:03.5241723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:03.5242262Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:03.5242768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:03.5243284Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:03.5337126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:03.5337698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:03.5338324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:03.5338844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:03.8921731Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:03.8929617Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:30:03.8966683Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2541 2022-09-27T15:30:03.8994262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2542 2022-09-27T15:30:03.9020672Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2543 2022-09-27T15:30:03.9048556Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2544 2022-09-27T15:30:05.0540325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:05.0540938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:05.0544246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:05.0544917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:05.0703014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:05.0703669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:05.0707793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:05.0708442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:05.0907488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:05.0908148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:05.0911753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:05.0912351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:05.1183261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:05.1183875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:05.1187950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:05.1188631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:05.1836418Z dist init r=2, world=4 2022-09-27T15:30:05.1982797Z dist init r=3, world=4 2022-09-27T15:30:05.2186222Z dist init r=0, world=4 2022-09-27T15:30:05.2451592Z dist init r=1, world=4 2022-09-27T15:30:05.2696190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:05.2796817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:05.2797251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:05.2798093Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:05.2798656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:05.2799338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:05.2799870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:05.2803798Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:05.2905227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:05.2905873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:05.2906402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:05.2906940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:05.6090637Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:05.6095570Z test_scatter_full_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-09-27T15:30:05.6133376Z Tests :meth:`scatter_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2597 2022-09-27T15:30:05.6159855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2598 2022-09-27T15:30:05.6187208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2599 2022-09-27T15:30:05.6214608Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2600 2022-09-27T15:30:06.7515517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:06.7516046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:06.7519899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:06.7520411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:06.7850639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:06.7851282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:06.7852065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:06.7852628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:06.7853377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:06.7853960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:06.7855159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:06.7855752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:06.7856560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:06.7857164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:06.7857942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:06.7858535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:06.8802708Z dist init r=3, world=4 2022-09-27T15:30:06.9140525Z dist init r=1, world=4 2022-09-27T15:30:06.9140894Z dist init r=2, world=4 2022-09-27T15:30:06.9228287Z dist init r=0, world=4 2022-09-27T15:30:06.9415870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:06.9515889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:06.9552664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:06.9553514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:06.9554135Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:06.9554868Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:06.9618334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:06.9618910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:06.9724937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:06.9725570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:06.9726211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:06.9726834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:07.3256372Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:07.3263086Z test_shard_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState) 2022-09-27T15:30:07.3301282Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2653 2022-09-27T15:30:07.3328188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2654 2022-09-27T15:30:07.3355933Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2655 2022-09-27T15:30:07.3383454Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2656 2022-09-27T15:30:08.4752447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:08.4753083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:08.4756136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:08.4756804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:08.4899663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:08.4900291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:08.4904561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:08.4906745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:08.5075340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:08.5075988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:08.5080025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:08.5080674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:08.5217739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:08.5218355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:08.5221914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:08.5222546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:08.6033477Z dist init r=1, world=4 2022-09-27T15:30:08.6183926Z dist init r=3, world=4 2022-09-27T15:30:08.6352730Z dist init r=0, world=4 2022-09-27T15:30:08.6484421Z dist init r=2, world=4 2022-09-27T15:30:08.6593741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:08.6694450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:08.6747250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:08.6747889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:08.6748683Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:08.6749210Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:08.6796931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:08.6797686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:08.6854676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:08.6855366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:08.6856026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:08.6856649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:09.0425525Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:09.0433490Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:30:09.0470639Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2709 2022-09-27T15:30:09.0497824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2710 2022-09-27T15:30:09.0525695Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2711 2022-09-27T15:30:09.0559300Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2712 2022-09-27T15:30:10.1602824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:10.1603491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:10.1607080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:10.1607719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:10.1612970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:10.1613586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:10.1618044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:10.1618628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:10.1632670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:10.1633263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:10.1637882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:10.1638656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:10.2263769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:10.2264373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:10.2267772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:10.2268475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:10.2889299Z dist init r=0, world=4 2022-09-27T15:30:10.2902861Z dist init r=3, world=4 2022-09-27T15:30:10.2915955Z dist init r=1, world=4 2022-09-27T15:30:10.3529709Z dist init r=2, world=4 2022-09-27T15:30:10.3716098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:10.3831817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:10.3933818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:10.3934522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:10.3935303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:10.3935833Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:10.3936363Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:10.4020031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:10.4041210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:10.4041880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:10.4044708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:10.4045461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:10.7600385Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:10.7608841Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:30:10.7646771Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2765 2022-09-27T15:30:10.7674386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2766 2022-09-27T15:30:10.7701467Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2767 2022-09-27T15:30:10.7729727Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2768 2022-09-27T15:30:11.9481355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:11.9481951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:11.9484754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:11.9485274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:11.9674743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:11.9675421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:11.9678972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:11.9679612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:11.9692741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:11.9693564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:11.9697510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:11.9698396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:11.9984414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:11.9985383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:11.9988904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:11.9989570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:12.0773133Z dist init r=0, world=4 2022-09-27T15:30:12.0970622Z dist init r=3, world=4 2022-09-27T15:30:12.0986438Z dist init r=2, world=4 2022-09-27T15:30:12.1264173Z dist init r=1, world=4 2022-09-27T15:30:12.1482464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:12.1483198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:12.1583626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:12.1584186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:12.1585078Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:12.1585788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:12.1586407Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:12.1587001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:12.1690885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:12.1691565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:12.1692083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:12.1692620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:12.4771256Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:12.4779144Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:30:12.4816194Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2821 2022-09-27T15:30:12.4844124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2822 2022-09-27T15:30:12.4871085Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2823 2022-09-27T15:30:12.4899094Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2824 2022-09-27T15:30:13.6321598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:13.6322341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:13.6325000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:13.6325712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:13.6390364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:13.6391198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:13.6394237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:13.6395120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:13.6564076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:13.6564838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:13.6567311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:13.6567914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:13.6598898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:13.6599505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:13.6603301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:13.6603905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:13.7614953Z dist init r=1, world=4 2022-09-27T15:30:13.7690836Z dist init r=3, world=4 2022-09-27T15:30:13.7851308Z dist init r=2, world=4 2022-09-27T15:30:13.7890212Z dist init r=0, world=4 2022-09-27T15:30:13.8127077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:13.8228068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:13.8228626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:13.8229152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:13.8229925Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:13.8230529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:13.8231067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:13.8231672Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:13.8336371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:13.8337015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:13.8337542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:13.8338052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:14.1939324Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:14.1947486Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:30:14.1982440Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2877 2022-09-27T15:30:14.2008704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2878 2022-09-27T15:30:14.2036756Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2879 2022-09-27T15:30:14.2070586Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2880 2022-09-27T15:30:15.3324875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:15.3325519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:15.3328493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:15.3329139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:15.3449643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:15.3450325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:15.3454348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:15.3454978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:15.3783132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:15.3783795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:15.3787608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:15.3788216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:15.3899043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:15.3899650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:15.3903039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:15.3903653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:15.4623183Z dist init r=1, world=4 2022-09-27T15:30:15.4753467Z dist init r=0, world=4 2022-09-27T15:30:15.5066593Z dist init r=2, world=4 2022-09-27T15:30:15.5173838Z dist init r=3, world=4 2022-09-27T15:30:15.5335315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:15.5376582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:15.5478323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:15.5478744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:15.5479446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:15.5480032Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:15.5480552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:15.5539432Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:15.5586700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:15.5587267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:15.5587770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:15.5588127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:15.9111665Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:15.9119600Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:30:15.9158142Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2933 2022-09-27T15:30:15.9185762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2934 2022-09-27T15:30:15.9212792Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2935 2022-09-27T15:30:15.9241751Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2936 2022-09-27T15:30:17.1141553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:17.1142082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:17.1144710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:17.1145391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:17.1173000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:17.1173543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:17.1177745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:17.1178363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:17.1287345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:17.1288010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:17.1293011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:17.1293733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:17.1485503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:17.1486139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:17.1489620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:17.1490250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:17.2439835Z dist init r=0, world=4 2022-09-27T15:30:17.2449391Z dist init r=1, world=4 2022-09-27T15:30:17.2565887Z dist init r=3, world=4 2022-09-27T15:30:17.2764911Z dist init r=2, world=4 2022-09-27T15:30:17.2877744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:17.3079849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:17.3181544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:17.3182383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:17.3183575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:17.3184156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:17.3184918Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:17.3185489Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:17.3189452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:17.3189944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:17.3190783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:17.3191136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:17.6283385Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:17.6291202Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:30:17.6329004Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2989 2022-09-27T15:30:17.6357759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2990 2022-09-27T15:30:17.6384635Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2991 2022-09-27T15:30:17.6412851Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2992 2022-09-27T15:30:18.8012934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:18.8013592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:18.8017442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:18.8018120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:18.8202949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:18.8203485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:18.8207249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:18.8207815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:18.8618199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:18.8618843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:18.8622277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:18.8622963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:18.8781381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:18.8781989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:18.8785436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:18.8786082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:18.9295026Z dist init r=2, world=4 2022-09-27T15:30:18.9502887Z dist init r=0, world=4 2022-09-27T15:30:18.9892447Z dist init r=3, world=4 2022-09-27T15:30:19.0053989Z dist init r=1, world=4 2022-09-27T15:30:19.0310816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:19.0410916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:19.0411610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:19.0412288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:19.0413352Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:19.0414309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:19.0415032Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:19.0415550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:19.0519572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:19.0520325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:19.0521333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:19.0521992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:19.3453970Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:19.3461039Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-09-27T15:30:19.3495021Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3045 2022-09-27T15:30:19.3522433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3046 2022-09-27T15:30:19.3549969Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3047 2022-09-27T15:30:19.3583836Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3048 2022-09-27T15:30:20.5000112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:20.5000732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:20.5004366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:20.5004986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:20.5303608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:20.5304142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:20.5306927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:20.5307422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:20.5557301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:20.5557952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:20.5561692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:20.5562289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:20.5633710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:20.5634365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:20.5639062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:20.5639696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:20.6284220Z dist init r=3, world=4 2022-09-27T15:30:20.6589522Z dist init r=2, world=4 2022-09-27T15:30:20.6852623Z dist init r=0, world=4 2022-09-27T15:30:20.6913059Z dist init r=1, world=4 2022-09-27T15:30:20.7201610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:20.7302954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:20.7405570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:20.7406499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:20.7406916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:20.7407416Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:20.7407922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:20.7408445Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:20.7512439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:20.7513536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:20.7514873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:20.7515514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:21.0625140Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:21.0632562Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-09-27T15:30:21.0670138Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3101 2022-09-27T15:30:21.0697842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3102 2022-09-27T15:30:21.0724729Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3103 2022-09-27T15:30:21.0752817Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3104 2022-09-27T15:30:22.1990222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:22.1990937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:22.1993555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:22.1994203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:22.2121433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:22.2121932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:22.2126083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:22.2126733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:22.2341315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:22.2341950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:22.2345426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:22.2346031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:22.2557450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:22.2558098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:22.2561186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:22.2561779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:22.3286492Z dist init r=3, world=4 2022-09-27T15:30:22.3407169Z dist init r=0, world=4 2022-09-27T15:30:22.3623751Z dist init r=2, world=4 2022-09-27T15:30:22.3849456Z dist init r=1, world=4 2022-09-27T15:30:22.4136349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:22.4237774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:22.4340017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:22.4340440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:22.4341092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:22.4341739Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:22.4342458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:22.4343043Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:22.4461186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:22.4461805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:22.4462256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:22.4478269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:22.7794794Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:22.7800313Z test_shard_full_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-09-27T15:30:22.7836953Z Tests :meth:`shard_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3157 2022-09-27T15:30:22.7863921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3158 2022-09-27T15:30:22.7891747Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3159 2022-09-27T15:30:22.7919107Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3160 2022-09-27T15:30:23.9146198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:23.9146832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:23.9150033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:23.9150678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:23.9281329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:23.9281837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:23.9285441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:23.9286087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:23.9495873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:23.9496485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:23.9499863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:23.9500499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:23.9762672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:23.9763306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:23.9766391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:23.9767019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:24.0442683Z dist init r=1, world=4 2022-09-27T15:30:24.0568107Z dist init r=2, world=4 2022-09-27T15:30:24.0775663Z dist init r=3, world=4 2022-09-27T15:30:24.1050294Z dist init r=0, world=4 2022-09-27T15:30:24.1185615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:24.1381573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:24.1483463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:24.1484095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:24.1484730Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:24.1485326Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:24.1485855Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:24.1489219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:24.1590882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:24.1591666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:24.1592329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:24.1593609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:24.4961491Z skip: Need at least 2 CUDA devices (1.716s) 2022-09-27T15:30:24.4968157Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState) 2022-09-27T15:30:24.5005526Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3213 2022-09-27T15:30:24.5033908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3214 2022-09-27T15:30:24.5062021Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3215 2022-09-27T15:30:24.5096986Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3216 2022-09-27T15:30:25.6304693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:25.6307807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:25.6308647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:25.6309291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:25.6618262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:25.6618905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:25.6621392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:25.6622005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:25.6922032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:25.6922717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:25.6927662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:25.6928271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:25.7207254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:25.7207872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:25.7211306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:25.7211956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:25.7597729Z dist init r=0, world=4 2022-09-27T15:30:25.7893250Z dist init r=1, world=4 2022-09-27T15:30:25.8195167Z dist init r=2, world=4 2022-09-27T15:30:25.8473861Z dist init r=3, world=4 2022-09-27T15:30:25.8606347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:25.8611748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:25.8713852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:25.8715216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:25.8715844Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:25.8716376Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:25.8716885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:25.8809879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:25.8821522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:25.8821929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:25.8822439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:25.8822962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:26.2137848Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T15:30:26.2144560Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState) 2022-09-27T15:30:26.2180991Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3269 2022-09-27T15:30:26.2208697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3270 2022-09-27T15:30:26.2235168Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3271 2022-09-27T15:30:26.2262560Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3272 2022-09-27T15:30:27.3931548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:27.3932173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:27.3933060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:27.3933804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:27.3934637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:27.3935190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:27.3935987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:27.3936581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:27.3973827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:27.3974420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:27.3978463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:27.3979081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:27.3981576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:27.3982239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:27.3986539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:27.3987176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:27.5225204Z dist init r=1, world=4 2022-09-27T15:30:27.5225572Z dist init r=3, world=4 2022-09-27T15:30:27.5261657Z dist init r=0, world=4 2022-09-27T15:30:27.5283478Z dist init r=2, world=4 2022-09-27T15:30:27.5534972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:27.5636627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:27.5637195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:27.5637690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:27.5638447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:27.5638991Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:27.5639524Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:27.5640041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:27.5744609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:27.5745180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:27.5745724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:27.5746242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:27.9304377Z skip: Need at least 2 CUDA devices (1.716s) 2022-09-27T15:30:27.9310851Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState) 2022-09-27T15:30:27.9348200Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3325 2022-09-27T15:30:27.9376099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3326 2022-09-27T15:30:27.9403276Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3327 2022-09-27T15:30:27.9431657Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3328 2022-09-27T15:30:29.0734449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:29.0735047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:29.0738108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:29.0738715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:29.0899075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:29.0902089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:29.0903012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:29.0903618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:29.1298308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:29.1298904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:29.1302592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:29.1303251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:29.1429122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:29.1429765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:29.1434266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:29.1435195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:29.2029777Z dist init r=1, world=4 2022-09-27T15:30:29.2184162Z dist init r=3, world=4 2022-09-27T15:30:29.2582035Z dist init r=2, world=4 2022-09-27T15:30:29.2713133Z dist init r=0, world=4 2022-09-27T15:30:29.2895984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:29.3099744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:29.3100327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:29.3101226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:29.3101649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:29.3102212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:29.3102826Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:29.3103432Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:29.3207102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:29.3207716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:29.3208233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:29.3208759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:29.6473957Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:29.6480378Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState) 2022-09-27T15:30:29.6517869Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3381 2022-09-27T15:30:29.6545652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3382 2022-09-27T15:30:29.6572198Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3383 2022-09-27T15:30:29.6600504Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3384 2022-09-27T15:30:30.8437727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:30.8438395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:30.8442622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:30.8506331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:30.8507221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:30.8508058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:30.8509128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:30.8509972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:30.8511055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:30.8511860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:30.8514803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:30.8515403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:30.8749599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:30.8750237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:30.8753661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:30.8754926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:30.9725974Z dist init r=1, world=4 2022-09-27T15:30:30.9793413Z dist init r=3, world=4 2022-09-27T15:30:30.9800017Z dist init r=0, world=4 2022-09-27T15:30:31.0029793Z dist init r=2, world=4 2022-09-27T15:30:31.0204774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:31.0304119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:31.0337121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:31.0337679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:31.0338300Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:31.0338833Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:31.0406963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:31.0407744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:31.0513256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:31.0514935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:31.0515597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:31.0516106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:31.3642475Z skip: Need at least 2 CUDA devices (1.717s) 2022-09-27T15:30:31.3642668Z 2022-09-27T15:30:31.3642966Z ---------------------------------------------------------------------- 2022-09-27T15:30:31.3643218Z Ran 50 tests in 84.574s 2022-09-27T15:30:31.3643338Z 2022-09-27T15:30:31.3643414Z OK (skipped=50) 2022-09-27T15:30:31.3643526Z 2022-09-27T15:30:31.3643620Z Generating XML reports... 2022-09-27T15:30:31.3741102Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20220927152906.xml 2022-09-27T15:30:31.6997504Z Running distributed/rpc/test_tensorpipe_agent ... [2022-09-27 15:30:31.699360] 2022-09-27T15:30:31.6998176Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:30:31.699489] 2022-09-27T15:30:32.9010602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp087kmyc6 2022-09-27T15:30:32.9011220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp087kmyc6/_remote_module_non_scriptable.py 2022-09-27T15:30:33.1614265Z , <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_comparison_uneven_inputs>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_dist_autograd_sparse_grads>]> 2022-09-27T15:30:33.1615487Z test_ddp_comparison (__main__.TensorPipeDdpComparisonTest) 2022-09-27T15:30:33.1615998Z test_ddp_comparison_uneven_inputs (__main__.TensorPipeDdpComparisonTest) 2022-09-27T15:30:33.1616616Z test_ddp_dist_autograd_local_vs_remote (__main__.TensorPipeDdpComparisonTest) 2022-09-27T15:30:33.1617223Z test_ddp_dist_autograd_sparse_grads (__main__.TensorPipeDdpComparisonTest) 2022-09-27T15:30:33.1618378Z , <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_outside>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_outside_uneven_inputs>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_no_ddp>]> 2022-09-27T15:30:33.1619392Z test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTest) 2022-09-27T15:30:33.1619932Z test_backward_ddp_outside (__main__.TensorPipeDdpUnderDistAutogradTest) 2022-09-27T15:30:33.1620521Z test_backward_ddp_outside_uneven_inputs (__main__.TensorPipeDdpUnderDistAutogradTest) 2022-09-27T15:30:33.1621110Z test_backward_no_ddp (__main__.TensorPipeDdpUnderDistAutogradTest) 2022-09-27T15:30:33.1630129Z , <__main__.TensorPipeDistAutogradTest testMethod=test_autograd_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_accumulate_grads>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_autograd_engine_error>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_complex_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_different_dtypes>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_different_tensor_dims>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_invalid_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_output_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_roots>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_round_trips>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_no_grad_on_tensor>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_node_failure>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_node_failure_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_python_udf_error>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref_multi>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref_nested>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_script_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_self>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_unused_send_function>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_unused_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_verify_hooks>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_without_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_without_rpc>, <__main__.TensorPipeDistAutogradTest testMethod=test_backwards_nested_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_clean_context_during_backward>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_nested_rpc>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_no_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_tensor_no_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_debug_info>, <__main__.TensorPipeDistAutogradTest testMethod=test_dist_autograd_profiling>, <__main__.TensorPipeDistAutogradTest testMethod=test_error_in_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_copy_sparse_indices_extra_ref>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_only_on_return_value>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_only_on_return_value_remote>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_builtin_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_builtin_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_call_itself>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_itself>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_python_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_python_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_mixed_requires_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_multiple_backward>, <__main__.TensorPipeDistAutogradTest testMethod=test_multiple_backward_with_errors>, <__main__.TensorPipeDistAutogradTest testMethod=test_nested_backward_accumulate_grads>, <__main__.TensorPipeDistAutogradTest testMethod=test_nested_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_grad_copy>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_grad_copy_sparse>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_remote>, <__main__.TensorPipeDistAutogradTest testMethod=test_post_hooks>, <__main__.TensorPipeDistAutogradTest testMethod=test_remote_complex_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_rpc_complex_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_thread_local_context_id>, <__main__.TensorPipeDistAutogradTest testMethod=test_trainer_ps>, <__main__.TensorPipeDistAutogradTest testMethod=test_trainer_ps_torchscript_functions>, <__main__.TensorPipeDistAutogradTest testMethod=test_worker_ids_recorded>]> 2022-09-27T15:30:33.1640301Z test_async_dist_autograd (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1640817Z test_autograd_context (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1641368Z test_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1641944Z test_backward_autograd_engine_error (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1642481Z test_backward_complex_python_udf (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1643087Z test_backward_different_dtypes (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1643659Z test_backward_different_tensor_dims (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1644208Z test_backward_invalid_args (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1644775Z test_backward_multiple_output_tensors (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1645371Z test_backward_multiple_roots (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1645929Z test_backward_multiple_round_trips (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1646485Z test_backward_no_grad_on_tensor (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1647053Z test_backward_node_failure (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1647640Z test_backward_node_failure_python_udf (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1648165Z test_backward_python_udf_error (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1648701Z test_backward_rref (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1649370Z test_backward_rref_multi (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1649873Z test_backward_rref_nested (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1650478Z test_backward_simple (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1651037Z test_backward_simple_python_udf (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1651617Z test_backward_simple_script_call (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1652109Z test_backward_simple_self (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1652659Z test_backward_unused_send_function (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1653219Z test_backward_unused_tensors (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1653742Z test_backward_verify_hooks (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1654290Z test_backward_without_context (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1654862Z test_backward_without_rpc (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1655412Z test_backwards_nested_python_udf (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1655983Z test_clean_context_during_backward (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1656307Z test_context_cleanup_nested_rpc (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1656621Z test_context_cleanup_no_tensors (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1656923Z test_context_cleanup_tensor_no_grad (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1657246Z test_context_cleanup_tensor_with_grad (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1657547Z test_debug_info (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1657829Z test_dist_autograd_profiling (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1658131Z test_error_in_context (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1658440Z test_grad_copy_sparse_indices_extra_ref (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1658763Z test_grad_only_on_return_value (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1659069Z test_grad_only_on_return_value_remote (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1659384Z test_graph_for_builtin_call (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1659693Z test_graph_for_builtin_remote_call (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1659993Z test_graph_for_py_nested_call (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1660305Z test_graph_for_py_nested_call_itself (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1660628Z test_graph_for_py_nested_remote_call (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1660943Z test_graph_for_py_nested_remote_call_itself (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1661263Z test_graph_for_python_call (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1661569Z test_graph_for_python_remote_call (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1661881Z test_mixed_requires_grad (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1662167Z test_multiple_backward (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1662475Z test_multiple_backward_with_errors (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1662801Z test_nested_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1663098Z test_nested_context (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1663383Z test_no_grad_copy (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1663675Z test_no_grad_copy_sparse (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1663994Z test_no_graph_with_tensors_not_require_grad (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1664326Z test_no_graph_with_tensors_not_require_grad_remote (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1664635Z test_post_hooks (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1664970Z test_remote_complex_args (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1665259Z test_rpc_complex_args (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1665556Z test_thread_local_context_id (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1665888Z test_trainer_ps (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1666183Z test_trainer_ps_torchscript_functions (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1666504Z test_worker_ids_recorded (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:30:33.1667146Z , <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_exception>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_exception_on_constructor>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_none_grads>]> 2022-09-27T15:30:33.1667756Z test_dist_optim (__main__.TensorPipeDistOptimizerTest) 2022-09-27T15:30:33.1668045Z test_dist_optim_exception (__main__.TensorPipeDistOptimizerTest) 2022-09-27T15:30:33.1668372Z test_dist_optim_exception_on_constructor (__main__.TensorPipeDistOptimizerTest) 2022-09-27T15:30:33.1668697Z test_dist_optim_none_grads (__main__.TensorPipeDistOptimizerTest) 2022-09-27T15:30:33.1669353Z , <__main__.TensorPipeJitDistAutogradTest testMethod=test_get_gradients>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_jit_fork_within_context>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_restore_context_after_swtich_to_jit_thread>]> 2022-09-27T15:30:33.1669966Z test_dist_backward (__main__.TensorPipeJitDistAutogradTest) 2022-09-27T15:30:33.1670265Z test_get_gradients (__main__.TensorPipeJitDistAutogradTest) 2022-09-27T15:30:33.1670578Z test_jit_fork_within_context (__main__.TensorPipeJitDistAutogradTest) 2022-09-27T15:30:33.1670921Z test_restore_context_after_swtich_to_jit_thread (__main__.TensorPipeJitDistAutogradTest) 2022-09-27T15:30:33.1676241Z , <__main__.TensorPipeJitRpcTest testMethod=test_all_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_args_and_kwargs_contain_different_types>, <__main__.TensorPipeJitRpcTest testMethod=test_args_kwargs_are_neither_passed>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_remote_multi>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_simple>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_decorator_order>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_return_type>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_return_type_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_async_script_throw>, <__main__.TensorPipeJitRpcTest testMethod=test_async_script_udf>, <__main__.TensorPipeJitRpcTest testMethod=test_call_fork_in_jit_with_profiling>, <__main__.TensorPipeJitRpcTest testMethod=test_call_python_function_remotely_from_script_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_call_rpc_with_profiling>, <__main__.TensorPipeJitRpcTest testMethod=test_call_script_function_that_not_exists_remotely_from_script>, <__main__.TensorPipeJitRpcTest testMethod=test_call_script_function_that_raises_remotely_from_script>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_chain>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_simple>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_with_exception>, <__main__.TensorPipeJitRpcTest testMethod=test_create_local_script_class_rref_in_py>, <__main__.TensorPipeJitRpcTest testMethod=test_create_local_script_module_rref_in_py>, <__main__.TensorPipeJitRpcTest testMethod=test_create_script_module_on_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_future_passed_between_python_and_jit>, <__main__.TensorPipeJitRpcTest testMethod=test_future_python_annotation>, <__main__.TensorPipeJitRpcTest testMethod=test_kwargs_not_passed>, <__main__.TensorPipeJitRpcTest testMethod=test_less_than_needed_args_are_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_load_script_module_with_pickled_rref>, <__main__.TensorPipeJitRpcTest testMethod=test_local_rref_local_value>, <__main__.TensorPipeJitRpcTest testMethod=test_more_than_needed_args_are_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_my_script_module_with_rrefs>, <__main__.TensorPipeJitRpcTest testMethod=test_no_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_record_function_jit_end_callbacks_with_fork>, <__main__.TensorPipeJitRpcTest testMethod=test_record_function_on_caller_rpc_async>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_module>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_throw>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_udf>, <__main__.TensorPipeJitRpcTest testMethod=test_return_local_script_class_rref_in_py_and_use_in_script>, <__main__.TensorPipeJitRpcTest testMethod=test_return_local_script_module_rref_in_py_and_use_in_script>, <__main__.TensorPipeJitRpcTest testMethod=test_rpc_async_jit_profiled>, <__main__.TensorPipeJitRpcTest testMethod=test_rpc_torchscript_record_function>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_as_arg_and_return>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_is_owner>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_jit_pickle_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_list_mutate>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_local_value>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_python_annotation>, <__main__.TensorPipeJitRpcTest testMethod=test_some_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_function>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_function_exception>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_functions_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_unexepected_kwarg_is_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_user_rrefs_confirmed>, <__main__.TensorPipeJitRpcTest testMethod=test_user_rrefs_confirmed_remote>]> 2022-09-27T15:30:33.1681076Z test_add_done_callback (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1681380Z test_all_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1681701Z test_args_and_kwargs_contain_different_types (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1682002Z test_args_kwargs_are_neither_passed (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1682293Z test_async_function_remote (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1682577Z test_async_function_remote_multi (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1682849Z test_async_function_simple (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1683143Z test_async_function_wrong_decorator_order (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1683455Z test_async_function_wrong_return_type (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1683768Z test_async_function_wrong_return_type_remote (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1684053Z test_async_script_throw (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1684322Z test_async_script_udf (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1684607Z test_call_fork_in_jit_with_profiling (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1684924Z test_call_python_function_remotely_from_script_not_supported (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1685244Z test_call_rpc_with_profiling (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1685566Z test_call_script_function_that_not_exists_remotely_from_script (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1686073Z test_call_script_function_that_raises_remotely_from_script (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1686365Z test_callback_chain (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1686661Z test_callback_simple (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1686940Z test_callback_with_exception (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1687276Z test_create_local_script_class_rref_in_py (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1687587Z test_create_local_script_module_rref_in_py (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1687894Z test_create_script_module_on_remote (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1688188Z test_future_passed_between_python_and_jit (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1688488Z test_future_python_annotation (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1688769Z test_kwargs_not_passed (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1689063Z test_less_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1689361Z test_load_script_module_with_pickled_rref (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1689656Z test_local_rref_local_value (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1689959Z test_more_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1690250Z test_my_script_module_with_rrefs (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1690551Z test_no_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1690909Z test_record_function_jit_end_callbacks_with_fork (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1691421Z test_record_function_on_caller_rpc_async (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1691864Z test_remote_script_module (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1692280Z test_remote_script_throw (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1692564Z test_remote_script_udf (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1692863Z test_return_local_script_class_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1693210Z test_return_local_script_module_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1693526Z test_rpc_async_jit_profiled (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1693808Z test_rpc_torchscript_record_function (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1694101Z test_rref_as_arg_and_return (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1694370Z test_rref_is_owner (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1694649Z test_rref_jit_pickle_not_supported (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1694918Z test_rref_list_mutate (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1695185Z test_rref_local_value (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1695456Z test_rref_python_annotation (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1695743Z test_some_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1696040Z test_torchscript_function (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1696331Z test_torchscript_function_exception (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1696632Z test_torchscript_functions_not_supported (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1696940Z test_unexepected_kwarg_is_specified (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1697227Z test_user_rrefs_confirmed (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1697508Z test_user_rrefs_confirmed_remote (__main__.TensorPipeJitRpcTest) 2022-09-27T15:30:33.1697878Z ]> 2022-09-27T15:30:33.1698278Z test_batch_updating_parameter_server (__main__.TensorPipeParameterServerTest) 2022-09-27T15:30:33.1698678Z ]> 2022-09-27T15:30:33.1699052Z test_rl_rpc (__main__.TensorPipeReinforcementLearningRpcTest) 2022-09-27T15:30:33.1700542Z , <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_async>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_async_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_sync>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_sync_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_with_kwargs>, <__main__.TensorPipeRemoteModuleTest testMethod=test_get_module_rref>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_module_py_pickle_not_supported>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_module_py_pickle_not_supported_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_parameters>, <__main__.TensorPipeRemoteModuleTest testMethod=test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire>, <__main__.TensorPipeRemoteModuleTest testMethod=test_train_eval>, <__main__.TensorPipeRemoteModuleTest testMethod=test_unsupported_methods>]> 2022-09-27T15:30:33.1702045Z test_bad_module (__main__.TensorPipeRemoteModuleTest) 2022-09-27T15:30:33.1702329Z test_forward_async (__main__.TensorPipeRemoteModuleTest) 2022-09-27T15:30:33.1702623Z test_forward_async_script (__main__.TensorPipeRemoteModuleTest) 2022-09-27T15:30:33.1702913Z test_forward_sync (__main__.TensorPipeRemoteModuleTest) 2022-09-27T15:30:33.1703193Z test_forward_sync_script (__main__.TensorPipeRemoteModuleTest) 2022-09-27T15:30:33.1703489Z test_forward_with_kwargs (__main__.TensorPipeRemoteModuleTest) 2022-09-27T15:30:33.1703780Z test_get_module_rref (__main__.TensorPipeRemoteModuleTest) 2022-09-27T15:30:33.1704082Z test_remote_module_py_pickle_not_supported (__main__.TensorPipeRemoteModuleTest) 2022-09-27T15:30:33.1704431Z test_remote_module_py_pickle_not_supported_script (__main__.TensorPipeRemoteModuleTest) 2022-09-27T15:30:33.1704756Z test_remote_parameters (__main__.TensorPipeRemoteModuleTest) 2022-09-27T15:30:33.1705093Z test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire (__main__.TensorPipeRemoteModuleTest) 2022-09-27T15:30:33.1705426Z test_train_eval (__main__.TensorPipeRemoteModuleTest) 2022-09-27T15:30:33.1705717Z test_unsupported_methods (__main__.TensorPipeRemoteModuleTest) 2022-09-27T15:30:33.1723240Z , <__main__.TensorPipeRpcTest testMethod=test_add_done_callback>, <__main__.TensorPipeRpcTest testMethod=test_add_with_id>, <__main__.TensorPipeRpcTest testMethod=test_all_gather>, <__main__.TensorPipeRpcTest testMethod=test_all_gather_timeout>, <__main__.TensorPipeRpcTest testMethod=test_async_add>, <__main__.TensorPipeRpcTest testMethod=test_async_class_method>, <__main__.TensorPipeRpcTest testMethod=test_async_class_method_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy_async>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_chained>, <__main__.TensorPipeRpcTest testMethod=test_async_function_chained_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_nested>, <__main__.TensorPipeRpcTest testMethod=test_async_function_nested_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_simple>, <__main__.TensorPipeRpcTest testMethod=test_async_function_with_future_ctor>, <__main__.TensorPipeRpcTest testMethod=test_async_function_with_future_ctor_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_cbs_jit_call>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_double_end_callbacks>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_double_end_callbacks_new_signatures>, <__main__.TensorPipeRpcTest testMethod=test_async_static_method>, <__main__.TensorPipeRpcTest testMethod=test_async_static_method_remote>, <__main__.TensorPipeRpcTest testMethod=test_build_rpc_profiling_key>, <__main__.TensorPipeRpcTest testMethod=test_builtin_remote_ret>, <__main__.TensorPipeRpcTest testMethod=test_builtin_remote_self>, <__main__.TensorPipeRpcTest testMethod=test_call_method_on_rref>, <__main__.TensorPipeRpcTest testMethod=test_callback_chain>, <__main__.TensorPipeRpcTest testMethod=test_callback_in_rpc>, <__main__.TensorPipeRpcTest testMethod=test_callback_multi>, <__main__.TensorPipeRpcTest testMethod=test_callback_none>, <__main__.TensorPipeRpcTest testMethod=test_callback_simple>, <__main__.TensorPipeRpcTest testMethod=test_callback_with_error>, <__main__.TensorPipeRpcTest testMethod=test_callback_with_ret>, <__main__.TensorPipeRpcTest testMethod=test_callback_wrong_arg_num>, <__main__.TensorPipeRpcTest testMethod=test_callback_wrong_arg_type>, <__main__.TensorPipeRpcTest testMethod=test_cannot_infer_backend_from_options>, <__main__.TensorPipeRpcTest testMethod=test_deadlock>, <__main__.TensorPipeRpcTest testMethod=test_debug_info>, <__main__.TensorPipeRpcTest testMethod=test_default_timeout_used>, <__main__.TensorPipeRpcTest testMethod=test_disable_gil_profiling>, <__main__.TensorPipeRpcTest testMethod=test_dist_init_decorator>, <__main__.TensorPipeRpcTest testMethod=test_duplicate_name>, <__main__.TensorPipeRpcTest testMethod=test_duplicate_name_2>, <__main__.TensorPipeRpcTest testMethod=test_expected_src>, <__main__.TensorPipeRpcTest testMethod=test_function_not_on_callee>, <__main__.TensorPipeRpcTest testMethod=test_future_done>, <__main__.TensorPipeRpcTest testMethod=test_future_done_exception>, <__main__.TensorPipeRpcTest testMethod=test_future_in_rpc>, <__main__.TensorPipeRpcTest testMethod=test_future_nested_callback>, <__main__.TensorPipeRpcTest testMethod=test_future_wait_twice>, <__main__.TensorPipeRpcTest testMethod=test_get_worker_infos>, <__main__.TensorPipeRpcTest testMethod=test_graceful_shutdown_with_uneven_workload>, <__main__.TensorPipeRpcTest testMethod=test_handle_send_exceptions>, <__main__.TensorPipeRpcTest testMethod=test_ignore_rref_leak>, <__main__.TensorPipeRpcTest testMethod=test_init_pg_then_rpc>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_then_pg>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_twice>, <__main__.TensorPipeRpcTest testMethod=test_int_callee>, <__main__.TensorPipeRpcTest testMethod=test_invalid_names>, <__main__.TensorPipeRpcTest testMethod=test_local_rref_no_fork>, <__main__.TensorPipeRpcTest testMethod=test_local_shutdown>, <__main__.TensorPipeRpcTest testMethod=test_local_shutdown_with_rpc>, <__main__.TensorPipeRpcTest testMethod=test_local_value_not_on_owner>, <__main__.TensorPipeRpcTest testMethod=test_mark_future_twice>, <__main__.TensorPipeRpcTest testMethod=test_multi_builtin_remote_ret>, <__main__.TensorPipeRpcTest testMethod=test_multi_layer_nested_async_rpc>, <__main__.TensorPipeRpcTest testMethod=test_multi_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_multi_rpc>, <__main__.TensorPipeRpcTest testMethod=test_my_parameter_server>, <__main__.TensorPipeRpcTest testMethod=test_nested_remote>, <__main__.TensorPipeRpcTest testMethod=test_nested_rpc>, <__main__.TensorPipeRpcTest testMethod=test_nested_rref>, <__main__.TensorPipeRpcTest testMethod=test_nested_rref_stress>, <__main__.TensorPipeRpcTest testMethod=test_non_cont_tensors>, <__main__.TensorPipeRpcTest testMethod=test_non_garbage_collected_user_rref_due_to_local_circular_dependency>, <__main__.TensorPipeRpcTest testMethod=test_nonzero>, <__main__.TensorPipeRpcTest testMethod=test_owner_equality>, <__main__.TensorPipeRpcTest testMethod=test_owner_rref_backward>, <__main__.TensorPipeRpcTest testMethod=test_pass_local_rrefs>, <__main__.TensorPipeRpcTest testMethod=test_pg_init_no_rpc_init>, <__main__.TensorPipeRpcTest testMethod=test_pickle_future>, <__main__.TensorPipeRpcTest testMethod=test_profiler_export_trace>, <__main__.TensorPipeRpcTest testMethod=test_profiler_remote_events_profiled>, <__main__.TensorPipeRpcTest testMethod=test_profiler_remote_events_profiled_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_key_names>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_memory>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_record_shapes>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_autograd_context>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_autograd_context_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_async_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_async_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_remote_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_remote_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_sync_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_sync_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_py_built_in>, <__main__.TensorPipeRpcTest testMethod=test_py_class_constructor>, <__main__.TensorPipeRpcTest testMethod=test_py_class_instance_method>, <__main__.TensorPipeRpcTest testMethod=test_py_class_method>, <__main__.TensorPipeRpcTest testMethod=test_py_class_static_method>, <__main__.TensorPipeRpcTest testMethod=test_py_function_exception>, <__main__.TensorPipeRpcTest testMethod=test_py_multi_async_call>, <__main__.TensorPipeRpcTest testMethod=test_py_nested_pickle>, <__main__.TensorPipeRpcTest testMethod=test_py_no_return_result>, <__main__.TensorPipeRpcTest testMethod=test_py_raise_in_user_func>, <__main__.TensorPipeRpcTest testMethod=test_py_raise_in_user_func_escaped_str>, <__main__.TensorPipeRpcTest testMethod=test_py_rpc_rref_args>, <__main__.TensorPipeRpcTest testMethod=test_py_rref_args>, <__main__.TensorPipeRpcTest testMethod=test_py_rref_args_user_share>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors_in_container>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors_multi_async_call>, <__main__.TensorPipeRpcTest testMethod=test_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_py_user_defined>, <__main__.TensorPipeRpcTest testMethod=test_register_rpc_backend_and_set_and_start_rpc_backend>, <__main__.TensorPipeRpcTest testMethod=test_reinit>, <__main__.TensorPipeRpcTest testMethod=test_remote_same_worker>, <__main__.TensorPipeRpcTest testMethod=test_remote_throw>, <__main__.TensorPipeRpcTest testMethod=test_remote_with_exception>, <__main__.TensorPipeRpcTest testMethod=test_return_future>, <__main__.TensorPipeRpcTest testMethod=test_return_future_async>, <__main__.TensorPipeRpcTest testMethod=test_return_future_remote>, <__main__.TensorPipeRpcTest testMethod=test_return_local_rrefs>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_all>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_multithreaded>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_partial_subset>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_subset>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_async_function>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_async_function_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_remote_record_function>, <__main__.TensorPipeRpcTest testMethod=test_rpc_return_rref>, <__main__.TensorPipeRpcTest testMethod=test_rpc_timeouts>, <__main__.TensorPipeRpcTest testMethod=test_rref_context_debug_info>, <__main__.TensorPipeRpcTest testMethod=test_rref_forward_chain>, <__main__.TensorPipeRpcTest testMethod=test_rref_get_future>, <__main__.TensorPipeRpcTest testMethod=test_rref_leak>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_class>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_class_self>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_non_exist>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_reuse>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_tensor>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_tensor_self>, <__main__.TensorPipeRpcTest testMethod=test_rref_py_pickle_not_supported>, <__main__.TensorPipeRpcTest testMethod=test_rref_str>, <__main__.TensorPipeRpcTest testMethod=test_rref_timeout>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_owner_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_owner_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_slow_init>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_with_error_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_with_error_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_scalar_add>, <__main__.TensorPipeRpcTest testMethod=test_self_add>, <__main__.TensorPipeRpcTest testMethod=test_self_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_remote_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_rpc_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_self_remote_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_self_rpc_arg>, <__main__.TensorPipeRpcTest testMethod=test_send_to_rank>, <__main__.TensorPipeRpcTest testMethod=test_server_process_global_profiler>, <__main__.TensorPipeRpcTest testMethod=test_set_and_get_default_rpc_timeout>, <__main__.TensorPipeRpcTest testMethod=test_shutdown_errors>, <__main__.TensorPipeRpcTest testMethod=test_shutdown_followed_by_rpc>, <__main__.TensorPipeRpcTest testMethod=test_stress_heavy_rpc>, <__main__.TensorPipeRpcTest testMethod=test_stress_heavy_rpc_torchscript>, <__main__.TensorPipeRpcTest testMethod=test_stress_light_rpc>, <__main__.TensorPipeRpcTest testMethod=test_use_rpc_pickler>, <__main__.TensorPipeRpcTest testMethod=test_use_rref_after_shutdown>, <__main__.TensorPipeRpcTest testMethod=test_user_rref_backward>, <__main__.TensorPipeRpcTest testMethod=test_user_rrefs_confirmed>, <__main__.TensorPipeRpcTest testMethod=test_user_rrefs_confirmed_remote>, <__main__.TensorPipeRpcTest testMethod=test_wait_all>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_builtin>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_python>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_script_function>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_multiple_call>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_raise_in_body>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_raise_in_user_func>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_timeout>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_with_exception>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_with_partial_exception>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_dense>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_timeout>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_twice_dense>, <__main__.TensorPipeRpcTest testMethod=test_worker_id>, <__main__.TensorPipeRpcTest testMethod=test_worker_info_pickle>, <__main__.TensorPipeRpcTest testMethod=test_world_size_one>, <__main__.TensorPipeRpcTest testMethod=test_wrong_types>]> 2022-09-27T15:30:33.1739675Z test_add (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1739930Z test_add_done_callback (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1740184Z test_add_with_id (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1740415Z test_all_gather (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1740670Z test_all_gather_timeout (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1740922Z test_async_add (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1741173Z test_async_class_method (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1741430Z test_async_class_method_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1741702Z test_async_class_rref_proxy (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1741976Z test_async_class_rref_proxy_async (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1742246Z test_async_class_rref_proxy_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1742520Z test_async_function_chained (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1742796Z test_async_function_chained_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1743063Z test_async_function_multi_chained (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1743356Z test_async_function_multi_chained_async (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1743654Z test_async_function_multi_chained_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1743943Z test_async_function_multi_fanout (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1744216Z test_async_function_multi_fanout_async (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1744512Z test_async_function_multi_fanout_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1744792Z test_async_function_nested (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1745053Z test_async_function_nested_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1745324Z test_async_function_raise (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1745594Z test_async_function_raise_async (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1745888Z test_async_function_raise_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1746161Z test_async_function_simple (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1746439Z test_async_function_with_future_ctor (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1746731Z test_async_function_with_future_ctor_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1747020Z test_async_function_wrong_return_type (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1747316Z test_async_function_wrong_return_type_async (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1747624Z test_async_function_wrong_return_type_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1747912Z test_async_record_function_cbs_jit_call (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1748217Z test_async_record_function_double_end_callbacks (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1748550Z test_async_record_function_double_end_callbacks_new_signatures (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1748856Z test_async_static_method (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1749114Z test_async_static_method_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1749385Z test_build_rpc_profiling_key (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1749653Z test_builtin_remote_ret (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1749938Z test_builtin_remote_self (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1750202Z test_call_method_on_rref (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1750452Z test_callback_chain (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1750722Z test_callback_in_rpc (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1750974Z test_callback_multi (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1751223Z test_callback_none (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1751531Z test_callback_simple (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1751792Z test_callback_with_error (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1752052Z test_callback_with_ret (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1752315Z test_callback_wrong_arg_num (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1752570Z test_callback_wrong_arg_type (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1752852Z test_cannot_infer_backend_from_options (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1753116Z test_deadlock (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1753346Z test_debug_info (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1753602Z test_default_timeout_used (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1753873Z test_disable_gil_profiling (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1754127Z test_dist_init_decorator (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1754383Z test_duplicate_name (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1754798Z test_duplicate_name_2 (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1755040Z test_expected_src (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1755299Z test_function_not_on_callee (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1755552Z test_future_done (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1755809Z test_future_done_exception (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1756054Z test_future_in_rpc (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1756309Z test_future_nested_callback (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1756571Z test_future_wait_twice (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1756814Z test_get_worker_infos (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1757095Z test_graceful_shutdown_with_uneven_workload (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1757388Z test_handle_send_exceptions (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1757638Z test_ignore_rref_leak (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1757895Z test_init_pg_then_rpc (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1758146Z test_init_rpc_then_pg (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1758383Z test_init_rpc_twice (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1758626Z test_int_callee (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1758870Z test_invalid_names (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1759121Z test_local_rref_no_fork (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1759364Z test_local_shutdown (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1759622Z test_local_shutdown_with_rpc (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1759895Z test_local_value_not_on_owner (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1760141Z test_mark_future_twice (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1760406Z test_multi_builtin_remote_ret (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1760683Z test_multi_layer_nested_async_rpc (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1760940Z test_multi_py_udf_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1761188Z test_multi_rpc (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1761439Z test_my_parameter_server (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1761680Z test_nested_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1761925Z test_nested_rpc (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1762165Z test_nested_rref (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1762416Z test_nested_rref_stress (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1762658Z test_non_cont_tensors (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1763039Z test_non_garbage_collected_user_rref_due_to_local_circular_dependency (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1763340Z test_nonzero (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1763632Z test_owner_equality (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1763889Z test_owner_rref_backward (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1764146Z test_pass_local_rrefs (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1764392Z test_pg_init_no_rpc_init (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1764645Z test_pickle_future (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1764905Z test_profiler_export_trace (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1765185Z test_profiler_remote_events_profiled (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1765482Z test_profiler_remote_events_profiled_single_threaded (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1765812Z test_profiler_rpc_key_names (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1766079Z test_profiler_rpc_memory (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1766342Z test_profiler_rpc_record_shapes (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1766627Z test_profiler_with_async_rpc_builtin (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1766938Z test_profiler_with_async_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1767231Z test_profiler_with_async_rpc_udf (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1767527Z test_profiler_with_async_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1767828Z test_profiler_with_autograd_context (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1768134Z test_profiler_with_autograd_context_single_threaded (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1768427Z test_profiler_with_remote_builtin (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1768727Z test_profiler_with_remote_builtin_single_threaded (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1769024Z test_profiler_with_remote_udf (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1769304Z test_profiler_with_remote_udf_single_threaded (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1769603Z test_profiler_with_script_async_rpc (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1769911Z test_profiler_with_script_async_rpc_single_threaded (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1770218Z test_profiler_with_script_remote_rpc (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1770514Z test_profiler_with_script_remote_rpc_single_threaded (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1770821Z test_profiler_with_script_sync_rpc (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1771122Z test_profiler_with_script_sync_rpc_single_threaded (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1771412Z test_profiler_with_sync_rpc_builtin (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1771716Z test_profiler_with_sync_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1772013Z test_profiler_with_sync_rpc_udf (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1772298Z test_profiler_with_sync_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1772575Z test_py_built_in (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1772831Z test_py_class_constructor (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1773106Z test_py_class_instance_method (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1773356Z test_py_class_method (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1773616Z test_py_class_static_method (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1773885Z test_py_function_exception (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1774136Z test_py_multi_async_call (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1774392Z test_py_nested_pickle (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1774652Z test_py_no_return_result (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1774899Z test_py_raise_in_user_func (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1775177Z test_py_raise_in_user_func_escaped_str (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1775449Z test_py_rpc_rref_args (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1775730Z test_py_rref_args (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1775979Z test_py_rref_args_user_share (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1776271Z test_py_tensors (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1776531Z test_py_tensors_in_container (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1776793Z test_py_tensors_multi_async_call (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1777052Z test_py_udf_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1777300Z test_py_user_defined (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1777584Z test_register_rpc_backend_and_set_and_start_rpc_backend (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1777865Z test_reinit (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1778113Z test_remote_same_worker (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1778352Z test_remote_throw (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1778609Z test_remote_with_exception (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1778868Z test_return_future (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1779121Z test_return_future_async (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1779371Z test_return_future_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1779632Z test_return_local_rrefs (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1779888Z test_rpc_barrier_all (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1780147Z test_rpc_barrier_multithreaded (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1780429Z test_rpc_barrier_partial_subset (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1780696Z test_rpc_barrier_subset (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1780956Z test_rpc_profiling_async_function (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1781260Z test_rpc_profiling_async_function_single_threaded (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1781576Z test_rpc_profiling_remote_record_function (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1781853Z test_rpc_return_rref (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1782093Z test_rpc_timeouts (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1782349Z test_rref_context_debug_info (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1782616Z test_rref_forward_chain (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1782858Z test_rref_get_future (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1783099Z test_rref_leak (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1783345Z test_rref_proxy_class (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1783593Z test_rref_proxy_class_self (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1783854Z test_rref_proxy_non_exist (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1784114Z test_rref_proxy_reuse (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1784357Z test_rref_proxy_tensor (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1784621Z test_rref_proxy_tensor_self (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1784898Z test_rref_py_pickle_not_supported (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1785158Z test_rref_str (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1785390Z test_rref_timeout (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1785645Z test_rref_type_blocking (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1785912Z test_rref_type_non_blocking (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1786169Z test_rref_type_owner_blocking (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1786446Z test_rref_type_owner_non_blocking (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1786719Z test_rref_type_slow_init (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1786978Z test_rref_type_with_error_blocking (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1787264Z test_rref_type_with_error_non_blocking (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1787532Z test_scalar_add (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1787758Z test_self_add (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1788005Z test_self_py_udf_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1788310Z test_self_remote_rref_as_remote_arg (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1788593Z test_self_remote_rref_as_rpc_arg (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1788865Z test_self_remote_rref_as_self_remote_arg (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1789184Z test_self_remote_rref_as_self_rpc_arg (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1789448Z test_send_to_rank (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1789704Z test_server_process_global_profiler (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1789992Z test_set_and_get_default_rpc_timeout (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1790260Z test_shutdown_errors (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1790515Z test_shutdown_followed_by_rpc (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1790779Z test_stress_heavy_rpc (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1791049Z test_stress_heavy_rpc_torchscript (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1791318Z test_stress_light_rpc (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1791661Z test_use_rpc_pickler (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1791924Z test_use_rref_after_shutdown (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1792191Z test_user_rref_backward (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1792442Z test_user_rrefs_confirmed (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1792714Z test_user_rrefs_confirmed_remote (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1792967Z test_wait_all (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1793212Z test_wait_all_exit_early_builtin (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1793487Z test_wait_all_exit_early_python (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1793772Z test_wait_all_exit_early_script_function (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1794054Z test_wait_all_multiple_call (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1794309Z test_wait_all_raise_in_body (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1794737Z test_wait_all_raise_in_user_func (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1795010Z test_wait_all_timeout (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1795263Z test_wait_all_with_exception (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1795545Z test_wait_all_with_partial_exception (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1795822Z test_wait_all_workers_dense (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1796084Z test_wait_all_workers_timeout (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1796357Z test_wait_all_workers_twice_dense (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1796614Z test_worker_id (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1796851Z test_worker_info_pickle (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1797104Z test_world_size_one (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1797349Z test_wrong_types (__main__.TensorPipeRpcTest) 2022-09-27T15:30:33.1801963Z , <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_multiple_round_trips_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_no_grad_on_tensor_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_multi_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_nested_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_python_udf_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_script_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_self_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backwards_nested_python_udf_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_nested_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_tensor_no_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_embedding_bag_with_no_grad_tensors>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_builtin_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_builtin_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_call_itself_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_itself_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_python_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_python_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_mixed_requires_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_multiple_backward_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_nested_backward_accumulate_grads_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_remote_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_remote_complex_args_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_rpc_complex_args_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_trainer_ps_sparse>]> 2022-09-27T15:30:33.1806512Z test_backward_different_dtypes_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1806936Z test_backward_multiple_round_trips_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1807352Z test_backward_no_grad_on_tensor_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1807740Z test_backward_rref_multi_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1808129Z test_backward_rref_nested_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1808510Z test_backward_rref_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1808899Z test_backward_simple_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1809296Z test_backward_simple_script_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1809694Z test_backward_simple_self_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1810078Z test_backward_simple_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1810461Z test_backwards_nested_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1810868Z test_context_cleanup_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1811275Z test_context_cleanup_tensor_no_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1811687Z test_context_cleanup_tensor_with_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1812089Z test_embedding_bag_with_no_grad_tensors (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1812644Z test_graph_for_builtin_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1813238Z test_graph_for_builtin_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1813924Z test_graph_for_py_nested_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1814642Z test_graph_for_py_nested_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1815242Z test_graph_for_py_nested_remote_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1815871Z test_graph_for_py_nested_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1816533Z test_graph_for_python_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1817231Z test_graph_for_python_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1817904Z test_mixed_requires_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1818596Z test_multiple_backward_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1819276Z test_nested_backward_accumulate_grads_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1820043Z test_no_graph_with_tensors_not_require_grad_remote_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1820941Z test_no_graph_with_tensors_not_require_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1821630Z test_remote_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1822173Z test_rpc_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1822558Z test_trainer_ps_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-09-27T15:30:33.1827358Z , <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_builtin_remote_self_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_and_static_init_rpc_together>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_existing_rank_can_communicate_with_new_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_init_rpc>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_init_rpc_without_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_new_rank_can_communicated_with_existing_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_infer_backend_from_options>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_mismatched_type_for_options>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_builtin_remote_ret_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_py_udf_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_my_parameter_server_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rref_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rref_stress_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_op_with_invalid_args>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rpc_rref_args_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rref_args_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rref_args_user_share_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_sparse_tensors_in_container>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_get_type_timeout_blocking>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_get_type_timeout_non_blocking>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_proxy_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_py_udf_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_remote_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_rpc_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_self_remote_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_self_rpc_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_send_to_rank_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_set_and_get_num_worker_threads>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_stress_heavy_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_tensorpipe_options_throw_on_timedelta_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_tensorpipe_set_default_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_wait_all_workers_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_wait_all_workers_twice_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_world_size_one_sparse>]> 2022-09-27T15:30:33.1832075Z test_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1832427Z test_builtin_remote_self_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1832788Z test_dynamic_and_static_init_rpc_together (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1833187Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1833593Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1833970Z test_dynamic_rpc_init_rpc (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1834322Z test_dynamic_rpc_init_rpc_without_rank (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1834906Z test_dynamic_rpc_new_rank_can_communicated_with_existing_rank (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1835291Z test_infer_backend_from_options (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1835650Z test_mismatched_type_for_options (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1836010Z test_multi_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1836348Z test_multi_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1836687Z test_multi_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1837026Z test_my_parameter_server_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1837355Z test_nested_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1837686Z test_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1838017Z test_nested_rref_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1838351Z test_nested_rref_stress_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1838676Z test_op_with_invalid_args (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1839013Z test_py_rpc_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1839346Z test_py_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1839674Z test_py_rref_args_user_share_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1840023Z test_py_sparse_tensors_in_container (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1840373Z test_rref_get_type_timeout_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1840715Z test_rref_get_type_timeout_non_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1841055Z test_rref_proxy_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1841385Z test_self_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1841804Z test_self_remote_rref_as_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1842157Z test_self_remote_rref_as_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1842568Z test_self_remote_rref_as_self_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1842940Z test_self_remote_rref_as_self_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1843276Z test_send_to_rank_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1843617Z test_set_and_get_num_worker_threads (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1843962Z test_stress_heavy_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1844329Z test_tensorpipe_options_throw_on_timedelta_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1844685Z test_tensorpipe_set_default_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1845027Z test_wait_all_workers_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1845369Z test_wait_all_workers_twice_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1845697Z test_world_size_one_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-09-27T15:30:33.1846392Z , <__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_send_remote_module_over_the_wire>, <__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_send_remote_module_over_the_wire_script_not_supported>]> 2022-09-27T15:30:33.1847110Z test_create_remote_module_from_module_rref (__main__.TensorPipeThreeWorkersRemoteModuleTest) 2022-09-27T15:30:33.1847499Z test_send_remote_module_over_the_wire (__main__.TensorPipeThreeWorkersRemoteModuleTest) 2022-09-27T15:30:33.1847909Z test_send_remote_module_over_the_wire_script_not_supported (__main__.TensorPipeThreeWorkersRemoteModuleTest) 2022-09-27T15:30:34.1953187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:34.1953862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:34.1962677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:34.1963279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:34.3536508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2cp5tf0 2022-09-27T15:30:34.3537653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2cp5tf0/_remote_module_non_scriptable.py 2022-09-27T15:30:34.6168882Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:30:34.6177333Z 2022-09-27T15:30:34.6177440Z Running tests... 2022-09-27T15:30:34.6178212Z ---------------------------------------------------------------------- 2022-09-27T15:30:34.8555638Z test_ddp_comparison (__main__.TensorPipeDdpComparisonTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/69662 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s) 2022-09-27T15:30:34.8556165Z 2022-09-27T15:30:34.8556374Z ---------------------------------------------------------------------- 2022-09-27T15:30:34.8556620Z Ran 1 test in 0.238s 2022-09-27T15:30:34.8556734Z 2022-09-27T15:30:34.8556808Z OK (skipped=1) 2022-09-27T15:30:34.8556904Z 2022-09-27T15:30:34.8556989Z Generating XML reports... 2022-09-27T15:30:34.8578958Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220927153034.xml 2022-09-27T15:30:36.1660826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:36.1661221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:36.1669668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:36.1670027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:36.3260742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm13l3_e7 2022-09-27T15:30:36.3261836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm13l3_e7/_remote_module_non_scriptable.py 2022-09-27T15:30:36.5861558Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:30:36.5870882Z 2022-09-27T15:30:36.5871014Z Running tests... 2022-09-27T15:30:36.5871510Z ---------------------------------------------------------------------- 2022-09-27T15:30:36.8209517Z test_ddp_comparison_uneven_inputs (__main__.TensorPipeDdpComparisonTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/72891 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.233s) 2022-09-27T15:30:36.8210034Z 2022-09-27T15:30:36.8210260Z ---------------------------------------------------------------------- 2022-09-27T15:30:36.8210508Z Ran 1 test in 0.234s 2022-09-27T15:30:36.8210622Z 2022-09-27T15:30:36.8210693Z OK (skipped=1) 2022-09-27T15:30:36.8210801Z 2022-09-27T15:30:36.8210874Z Generating XML reports... 2022-09-27T15:30:36.8232772Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220927153036.xml 2022-09-27T15:30:38.1233254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:38.1233665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:38.1241811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:38.1242192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:38.2846548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpln5nqa13 2022-09-27T15:30:38.2847693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpln5nqa13/_remote_module_non_scriptable.py 2022-09-27T15:30:38.5494027Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:30:38.5503539Z 2022-09-27T15:30:38.5503665Z Running tests... 2022-09-27T15:30:38.5504073Z ---------------------------------------------------------------------- 2022-09-27T15:30:38.7941556Z test_ddp_dist_autograd_local_vs_remote (__main__.TensorPipeDdpComparisonTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3478 2022-09-27T15:30:38.7968758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3479 2022-09-27T15:30:38.7995230Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3480 2022-09-27T15:30:38.8022691Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3481 2022-09-27T15:30:39.9387674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:39.9388338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:39.9623116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:39.9623543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:39.9664166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:39.9665114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:39.9670280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:39.9671114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:39.9883068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:39.9883600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:40.0015692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:40.0016478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:40.0051704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:40.0052338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:40.0055629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:40.0056194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:40.1153313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tgdjbo5 2022-09-27T15:30:40.1154025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tgdjbo5/_remote_module_non_scriptable.py 2022-09-27T15:30:40.1273782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjv1z210a 2022-09-27T15:30:40.1275180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjv1z210a/_remote_module_non_scriptable.py 2022-09-27T15:30:40.1315442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7a_nznm 2022-09-27T15:30:40.1317096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7a_nznm/_remote_module_non_scriptable.py 2022-09-27T15:30:40.1323925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwi6r00eu 2022-09-27T15:30:40.1325887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwi6r00eu/_remote_module_non_scriptable.py 2022-09-27T15:30:40.3736271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:40.3837530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:40.3847598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:40.3913658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:40.6600389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:40.6701338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:40.6726611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:40.6727879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:40.6729125Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:40.6805067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:40.6806230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:40.6807356Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:40.7262777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:40.7264273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:40.7265284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:40.7272689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:40.7471478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:40.7472295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:40.7473026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:40.7480017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:41.2081700Z ok (2.657s) 2022-09-27T15:30:41.2082028Z 2022-09-27T15:30:41.2082417Z ---------------------------------------------------------------------- 2022-09-27T15:30:41.2082672Z Ran 1 test in 2.658s 2022-09-27T15:30:41.2082787Z 2022-09-27T15:30:41.2082842Z OK 2022-09-27T15:30:41.2082978Z 2022-09-27T15:30:41.2083085Z Generating XML reports... 2022-09-27T15:30:41.2118141Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220927153038.xml 2022-09-27T15:30:42.5727911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:42.5728536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:42.5737354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:42.5737981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:42.7322207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsc6cvsi 2022-09-27T15:30:42.7322956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsc6cvsi/_remote_module_non_scriptable.py 2022-09-27T15:30:42.9950086Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:30:42.9960518Z 2022-09-27T15:30:42.9960996Z Running tests... 2022-09-27T15:30:42.9961402Z ---------------------------------------------------------------------- 2022-09-27T15:30:43.2368661Z test_ddp_dist_autograd_sparse_grads (__main__.TensorPipeDdpComparisonTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3733 2022-09-27T15:30:43.2394901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3734 2022-09-27T15:30:43.2421158Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3735 2022-09-27T15:30:43.2448649Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3736 2022-09-27T15:30:44.3132730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:44.3133424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:44.3431763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:44.3432435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:44.3433214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:44.3433793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:44.3491553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:44.3492126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:44.3708761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:44.3709388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:44.3813749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:44.3814531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:44.3815368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:44.3815946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:44.4087799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:44.4088493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:44.4768833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhruiax6 2022-09-27T15:30:44.4769659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhruiax6/_remote_module_non_scriptable.py 2022-09-27T15:30:44.5087328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03_vq4l1 2022-09-27T15:30:44.5088155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkyue9dgq 2022-09-27T15:30:44.5088941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03_vq4l1/_remote_module_non_scriptable.py 2022-09-27T15:30:44.5089685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkyue9dgq/_remote_module_non_scriptable.py 2022-09-27T15:30:44.5359879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2c3rnkc 2022-09-27T15:30:44.5361166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2c3rnkc/_remote_module_non_scriptable.py 2022-09-27T15:30:44.7343051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:44.7651707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:44.7652162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:44.7982802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:45.0261512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:45.0361491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:45.0463410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:45.0464257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:45.0465509Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:45.0466642Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:45.0468048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:45.0469504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:30:45.1151554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:45.1155857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:45.1156429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:45.1241525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:45.5506476Z ok (2.554s) 2022-09-27T15:30:45.5506951Z 2022-09-27T15:30:45.5507520Z ---------------------------------------------------------------------- 2022-09-27T15:30:45.5507880Z Ran 1 test in 2.555s 2022-09-27T15:30:45.5508037Z 2022-09-27T15:30:45.5515416Z OK 2022-09-27T15:30:45.5515910Z 2022-09-27T15:30:45.5516099Z Generating XML reports... 2022-09-27T15:30:45.5545223Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220927153042.xml 2022-09-27T15:30:46.9353570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:46.9353946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:46.9362486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:46.9362846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:47.0947274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdfmug4wg 2022-09-27T15:30:47.0948404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdfmug4wg/_remote_module_non_scriptable.py 2022-09-27T15:30:47.3647502Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:30:47.3657256Z 2022-09-27T15:30:47.3657728Z Running tests... 2022-09-27T15:30:47.3658176Z ---------------------------------------------------------------------- 2022-09-27T15:30:47.6047517Z test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4000 2022-09-27T15:30:47.6074125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4001 2022-09-27T15:30:47.6100588Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4002 2022-09-27T15:30:47.6128351Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4003 2022-09-27T15:30:47.6156643Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 4004 2022-09-27T15:30:47.6192126Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 4005 2022-09-27T15:30:48.9126773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:48.9127467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:48.9128259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:48.9128895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:48.9532536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:48.9533168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:48.9536656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:48.9537209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:49.0823157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rlmv0zx 2022-09-27T15:30:49.0824385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rlmv0zx/_remote_module_non_scriptable.py 2022-09-27T15:30:49.0828128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppnvidg4t 2022-09-27T15:30:49.0830368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppnvidg4t/_remote_module_non_scriptable.py 2022-09-27T15:30:49.3073062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:49.3073715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:49.3486416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:49.3598636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:49.3599040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:49.3667318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:49.3957838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:49.3958229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:49.4144300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:49.4144644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:49.4152917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:49.4153320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:49.4455630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:49.4456253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:49.4678688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:49.4679397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:49.4685077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:49.4685750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:49.5104558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34j0cta5 2022-09-27T15:30:49.5105697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34j0cta5/_remote_module_non_scriptable.py 2022-09-27T15:30:49.5733570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqz0pldu9 2022-09-27T15:30:49.5734358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqz0pldu9/_remote_module_non_scriptable.py 2022-09-27T15:30:49.6461384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbvk5jasm 2022-09-27T15:30:49.6462626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ifn6e4e 2022-09-27T15:30:49.6463344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbvk5jasm/_remote_module_non_scriptable.py 2022-09-27T15:30:49.6464003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ifn6e4e/_remote_module_non_scriptable.py 2022-09-27T15:30:49.7732088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:49.8360550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-09-27T15:30:49.9051228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-09-27T15:30:49.9075419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:50.2327902Z 2022-09-27 15:30:50,231 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0... 2022-09-27T15:30:50.2329460Z 2022-09-27 15:30:50,231 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3] 2022-09-27T15:30:50.2429946Z 2022-09-27 15:30:50,241 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1... 2022-09-27T15:30:50.2431112Z 2022-09-27 15:30:50,242 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3] 2022-09-27T15:30:50.2531132Z 2022-09-27 15:30:50,252 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2... 2022-09-27T15:30:50.2532278Z 2022-09-27 15:30:50,252 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3] 2022-09-27T15:30:50.2627076Z 2022-09-27 15:30:50,261 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3... 2022-09-27T15:30:50.2628234Z 2022-09-27 15:30:50,261 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process... 2022-09-27T15:30:50.2635283Z 2022-09-27 15:30:50,262 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3] 2022-09-27T15:30:50.2646180Z 2022-09-27 15:30:50,263 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running. 2022-09-27T15:30:50.2859174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:50.2960419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:50.2967825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:50.2973300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:50.3074064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-09-27T15:30:50.3086070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-09-27T15:30:50.3087355Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:30:50.3088501Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:30:50.3163989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:30:50.3173510Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:30:50.3187182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:30:50.3188367Z 2022-09-27 15:30:50,317 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0... 2022-09-27T15:30:50.3189491Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:30:50.3190334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-09-27T15:30:50.3195041Z 2022-09-27 15:30:50,318 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2... 2022-09-27T15:30:50.3207356Z 2022-09-27 15:30:50,319 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master 2022-09-27T15:30:50.3215563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-09-27T15:30:50.3268179Z 2022-09-27 15:30:50,326 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1... 2022-09-27T15:30:50.3276820Z 2022-09-27 15:30:50,327 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3... 2022-09-27T15:30:50.3524690Z 2022-09-27 15:30:50,351 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3 2022-09-27T15:30:50.3533342Z 2022-09-27 15:30:50,351 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3 2022-09-27T15:30:50.4985524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-09-27T15:30:50.5085249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-09-27T15:30:50.5195313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-09-27T15:30:50.5196212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-09-27T15:30:50.5197131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:30:50.5198030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:30:50.5198874Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:30:50.5202702Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:30:50.5203560Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:30:50.5214083Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:30:50.5214930Z 2022-09-27 15:30:50,520 ddp_under_dist_autograd_test.py:153 INFO p:process 2 t:Dummy-2: Use DDP for the second local net. 2022-09-27T15:30:50.5219990Z 2022-09-27 15:30:50,521 ddp_under_dist_autograd_test.py:153 INFO p:process 0 t:Dummy-2: Use DDP for the second local net. 2022-09-27T15:30:50.5246304Z 2022-09-27 15:30:50,524 ddp_under_dist_autograd_test.py:153 INFO p:process 3 t:Dummy-2: Use DDP for the second local net. 2022-09-27T15:30:50.5306622Z 2022-09-27 15:30:50,530 ddp_under_dist_autograd_test.py:153 INFO p:process 1 t:Dummy-2: Use DDP for the second local net. 2022-09-27T15:30:50.5505123Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-09-27T15:30:50.5506171Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-09-27T15:30:50.5530619Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-09-27T15:30:50.5531594Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-09-27T15:30:50.5546382Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-09-27T15:30:50.5547321Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-09-27T15:30:50.5581716Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-09-27T15:30:50.5582688Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-09-27T15:30:50.5612833Z 2022-09-27 15:30:50,560 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:30:50.5613642Z 2022-09-27 15:30:50,560 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:30:50.5614395Z 2022-09-27 15:30:50,560 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:30:50.5615304Z 2022-09-27 15:30:50,560 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params. 2022-09-27T15:30:50.5616255Z 2022-09-27 15:30:50,560 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params. 2022-09-27T15:30:50.5617217Z 2022-09-27 15:30:50,560 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params. 2022-09-27T15:30:50.5657979Z 2022-09-27 15:30:50,565 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:30:50.5659252Z 2022-09-27 15:30:50,565 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params. 2022-09-27T15:30:50.6137027Z 2022-09-27 15:30:50,613 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:30:50.6137554Z [ 1., 1.], 2022-09-27T15:30:50.6137828Z [ 1., -1.], 2022-09-27T15:30:50.6138443Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:50.6139061Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:50.6139444Z tensor([[-1., 1.], 2022-09-27T15:30:50.6139739Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-09-27T15:30:50.6140037Z [ 0., 16.]])} 2022-09-27T15:30:50.6142043Z 2022-09-27 15:30:50,613 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-09-27T15:30:50.6142545Z [ 1., -1.], 2022-09-27T15:30:50.6142802Z [ 1., 1.], 2022-09-27T15:30:50.6143417Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:50.6144153Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:50.6144685Z tensor([[-1., 1.], 2022-09-27T15:30:50.6145252Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-09-27T15:30:50.6145738Z [ 0., -16.]])} 2022-09-27T15:30:50.6146468Z 2022-09-27 15:30:50,613 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-09-27T15:30:50.6229547Z [-1., -1.], 2022-09-27T15:30:50.6230085Z [-1., 1.], 2022-09-27T15:30:50.6230930Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:50.6231674Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:50.6232191Z tensor([[-1., 1.], 2022-09-27T15:30:50.6232767Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-09-27T15:30:50.6233277Z [ 16., -16.]])} 2022-09-27T15:30:50.6234209Z 2022-09-27 15:30:50,617 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:30:50.6234942Z [-1., 1.], 2022-09-27T15:30:50.6235316Z [-1., -1.], 2022-09-27T15:30:50.6235978Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:50.6236855Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:50.6237417Z tensor([[-1., 1.], 2022-09-27T15:30:50.6238000Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-09-27T15:30:50.6239402Z [-16., 16.]])} 2022-09-27T15:30:50.6335057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:50.6368472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:50.6369279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:50.6385426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:50.6433476Z 2022-09-27 15:30:50,641 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-09-27T15:30:50.6433813Z [ 1., -1.], 2022-09-27T15:30:50.6434008Z [ 1., 1.], 2022-09-27T15:30:50.6434842Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:50.6435334Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:50.6435601Z tensor([[-1., 1.], 2022-09-27T15:30:50.6435875Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-09-27T15:30:50.6436109Z [ 0., -16.]])} 2022-09-27T15:30:50.6436532Z 2022-09-27 15:30:50,641 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-09-27T15:30:50.6436856Z [-1., -1.], 2022-09-27T15:30:50.6437045Z [-1., 1.], 2022-09-27T15:30:50.6437420Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:50.6437840Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:50.6438100Z tensor([[-1., 1.], 2022-09-27T15:30:50.6438352Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-09-27T15:30:50.6438584Z [ 16., -16.]])} 2022-09-27T15:30:50.6439009Z 2022-09-27 15:30:50,642 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:30:50.6439298Z [ 1., 1.], 2022-09-27T15:30:50.6439574Z [ 1., -1.], 2022-09-27T15:30:50.6439982Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:50.6440655Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:50.6441071Z tensor([[-1., 1.], 2022-09-27T15:30:50.6441400Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-09-27T15:30:50.6441601Z [ 0., 16.]])} 2022-09-27T15:30:50.6442038Z 2022-09-27 15:30:50,643 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:30:50.6442344Z [-1., 1.], 2022-09-27T15:30:50.6442535Z [-1., -1.], 2022-09-27T15:30:50.6445351Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:50.6446022Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:50.6446309Z tensor([[-1., 1.], 2022-09-27T15:30:50.6446707Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-09-27T15:30:50.6449184Z [-16., 16.]])} 2022-09-27T15:30:50.6630202Z 2022-09-27 15:30:50,662 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:30:50.6630774Z [-1., 1.], 2022-09-27T15:30:50.6631107Z [-1., -1.], 2022-09-27T15:30:50.6631775Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:50.6632450Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:50.6635469Z tensor([[-1., 1.], 2022-09-27T15:30:50.6639484Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-09-27T15:30:50.6640126Z [-16., 16.]])} 2022-09-27T15:30:50.6641918Z 2022-09-27 15:30:50,662 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-09-27T15:30:50.6642596Z [-1., -1.], 2022-09-27T15:30:50.6643475Z [-1., 1.], 2022-09-27T15:30:50.6645077Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:50.6646355Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:50.6648421Z tensor([[-1., 1.], 2022-09-27T15:30:50.6650346Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-09-27T15:30:50.6650853Z [ 16., -16.]])} 2022-09-27T15:30:50.6652659Z 2022-09-27 15:30:50,662 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:30:50.6653662Z [ 1., 1.], 2022-09-27T15:30:50.6658628Z [ 1., -1.], 2022-09-27T15:30:50.6660979Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:50.6663293Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:50.6665309Z tensor([[-1., 1.], 2022-09-27T15:30:50.6667192Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-09-27T15:30:50.6668972Z [ 0., 16.]])} 2022-09-27T15:30:50.6671754Z 2022-09-27 15:30:50,665 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-09-27T15:30:50.6674020Z [ 1., -1.], 2022-09-27T15:30:50.6675900Z [ 1., 1.], 2022-09-27T15:30:50.6678320Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:50.6680551Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:50.6682411Z tensor([[-1., 1.], 2022-09-27T15:30:50.6684097Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-09-27T15:30:50.6685871Z [ 0., -16.]])} 2022-09-27T15:30:50.6732263Z 2022-09-27 15:30:50,672 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0... 2022-09-27T15:30:50.6737141Z 2022-09-27 15:30:50,673 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1... 2022-09-27T15:30:50.6741375Z 2022-09-27 15:30:50,673 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2... 2022-09-27T15:30:50.6750670Z 2022-09-27 15:30:50,674 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3... 2022-09-27T15:30:50.6758633Z 2022-09-27 15:30:50,675 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker. 2022-09-27T15:30:51.2279781Z ok (3.862s) 2022-09-27T15:30:51.2279995Z 2022-09-27T15:30:51.2283073Z ---------------------------------------------------------------------- 2022-09-27T15:30:51.2283458Z Ran 1 test in 3.862s 2022-09-27T15:30:51.2283631Z 2022-09-27T15:30:51.2283727Z OK 2022-09-27T15:30:51.2283883Z 2022-09-27T15:30:51.2284030Z Generating XML reports... 2022-09-27T15:30:51.2317587Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220927153047.xml 2022-09-27T15:30:52.6314802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:52.6315477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:52.6323901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:52.6324501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:52.7902203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckgh76i9 2022-09-27T15:30:52.7902910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckgh76i9/_remote_module_non_scriptable.py 2022-09-27T15:30:53.0495708Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:30:53.0505515Z 2022-09-27T15:30:53.0505980Z Running tests... 2022-09-27T15:30:53.0506595Z ---------------------------------------------------------------------- 2022-09-27T15:30:53.2892027Z test_backward_ddp_outside (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4414 2022-09-27T15:30:53.2919164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4415 2022-09-27T15:30:53.2945324Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4416 2022-09-27T15:30:53.2973589Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4417 2022-09-27T15:30:53.3000757Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 4418 2022-09-27T15:30:53.3038414Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 4419 2022-09-27T15:30:54.7223784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:54.7224430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:54.7356650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:54.7357376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:54.7622359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:54.7623143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:54.7744996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:54.7745672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:54.8343280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:54.8344280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:54.8741768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:54.8742431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:54.8829673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:54.8830367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:54.8905402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pfd4rjt 2022-09-27T15:30:54.8907022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pfd4rjt/_remote_module_non_scriptable.py 2022-09-27T15:30:54.9013607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_noa5nz 2022-09-27T15:30:54.9015492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_noa5nz/_remote_module_non_scriptable.py 2022-09-27T15:30:54.9245668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:54.9246358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:55.0156831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:55.0157491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:55.0667466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:55.0668158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:55.0809506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:55.0810152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:55.0825542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_vs7kug 2022-09-27T15:30:55.0828086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_vs7kug/_remote_module_non_scriptable.py 2022-09-27T15:30:55.1278786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2o7htfz5 2022-09-27T15:30:55.1279743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2o7htfz5/_remote_module_non_scriptable.py 2022-09-27T15:30:55.1312778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:55.1313376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:55.1502589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:30:55.1652512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:30:55.2513836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4dvzr7cs 2022-09-27T15:30:55.2515382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4dvzr7cs/_remote_module_non_scriptable.py 2022-09-27T15:30:55.2932128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmtowqpd5 2022-09-27T15:30:55.2935011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmtowqpd5/_remote_module_non_scriptable.py 2022-09-27T15:30:55.3868680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-09-27T15:30:55.4149303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:30:55.5099254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-09-27T15:30:55.5541566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:30:55.8547366Z 2022-09-27 15:30:55,853 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0... 2022-09-27T15:30:55.8548728Z 2022-09-27 15:30:55,853 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3] 2022-09-27T15:30:55.8643846Z 2022-09-27 15:30:55,863 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1... 2022-09-27T15:30:55.8649527Z 2022-09-27 15:30:55,864 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3] 2022-09-27T15:30:55.8743871Z 2022-09-27 15:30:55,873 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2... 2022-09-27T15:30:55.8749128Z 2022-09-27 15:30:55,874 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3] 2022-09-27T15:30:55.8943089Z 2022-09-27 15:30:55,893 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running. 2022-09-27T15:30:55.8952910Z 2022-09-27 15:30:55,894 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process... 2022-09-27T15:30:55.8960323Z 2022-09-27 15:30:55,893 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3... 2022-09-27T15:30:55.8961808Z 2022-09-27 15:30:55,895 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3] 2022-09-27T15:30:55.9081228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:30:55.9082959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:30:55.9380721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:30:55.9483814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:30:55.9604063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-09-27T15:30:55.9604896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-09-27T15:30:55.9606310Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:30:55.9607446Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:30:55.9608573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:30:55.9609676Z 2022-09-27 15:30:55,959 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0... 2022-09-27T15:30:55.9610768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:30:55.9611868Z 2022-09-27 15:30:55,959 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1... 2022-09-27T15:30:55.9612951Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:30:55.9614080Z 2022-09-27 15:30:55,960 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3... 2022-09-27T15:30:55.9615148Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:30:55.9626071Z 2022-09-27 15:30:55,962 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master 2022-09-27T15:30:55.9644733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-09-27T15:30:55.9693492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-09-27T15:30:55.9698076Z 2022-09-27 15:30:55,969 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2... 2022-09-27T15:30:55.9959703Z 2022-09-27 15:30:55,995 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3 2022-09-27T15:30:55.9965390Z 2022-09-27 15:30:55,995 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3 2022-09-27T15:30:56.1098474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-09-27T15:30:56.1225686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-09-27T15:30:56.1301059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-09-27T15:30:56.1301937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-09-27T15:30:56.1304855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:30:56.1313223Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:30:56.1317148Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:30:56.1321047Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:30:56.1333142Z 2022-09-27 15:30:56,132 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:30:56.1336567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:30:56.1346036Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:30:56.1353123Z 2022-09-27 15:30:56,134 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:30:56.1356187Z 2022-09-27 15:30:56,135 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:30:56.1358560Z 2022-09-27 15:30:56,135 ddp_under_dist_autograd_test.py:202 INFO p:process 1 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-09-27T15:30:56.1377805Z 2022-09-27 15:30:56,137 ddp_under_dist_autograd_test.py:202 INFO p:process 0 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-09-27T15:30:56.1389407Z 2022-09-27 15:30:56,138 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:30:56.1391088Z 2022-09-27 15:30:56,138 ddp_under_dist_autograd_test.py:202 INFO p:process 2 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-09-27T15:30:56.1392981Z 2022-09-27 15:30:56,139 ddp_under_dist_autograd_test.py:202 INFO p:process 3 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-09-27T15:30:56.1651077Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-09-27T15:30:56.1652808Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-09-27T15:30:56.1688444Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-09-27T15:30:56.1689191Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-09-27T15:30:56.1747555Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-09-27T15:30:56.1748236Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-09-27T15:30:56.1764127Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-09-27T15:30:56.1764829Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-09-27T15:30:56.1790155Z 2022-09-27 15:30:56,178 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-09-27T15:30:56.1791112Z 2022-09-27 15:30:56,178 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-09-27T15:30:56.1792136Z 2022-09-27 15:30:56,178 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-09-27T15:30:56.1796481Z 2022-09-27 15:30:56,179 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-09-27T15:30:56.2489610Z 2022-09-27 15:30:56,248 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:30:56.2490299Z [ 1., 1.], 2022-09-27T15:30:56.2490733Z [ 1., -1.], 2022-09-27T15:30:56.2491539Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:56.2492218Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:56.2492602Z tensor([[-1., 1.], 2022-09-27T15:30:56.2492895Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-09-27T15:30:56.2493188Z [0., 0.]])} 2022-09-27T15:30:56.2502558Z 2022-09-27 15:30:56,249 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:30:56.2503628Z [-1., 1.], 2022-09-27T15:30:56.2504020Z [-1., -1.], 2022-09-27T15:30:56.2504843Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:56.2505830Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:56.2506352Z tensor([[-1., 1.], 2022-09-27T15:30:56.2506792Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-09-27T15:30:56.2507211Z [0., 0.]])} 2022-09-27T15:30:56.2520275Z 2022-09-27 15:30:56,251 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-09-27T15:30:56.2521247Z [-1., -1.], 2022-09-27T15:30:56.2522245Z [-1., 1.], 2022-09-27T15:30:56.2523456Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:56.2524680Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:56.2525264Z tensor([[-1., 1.], 2022-09-27T15:30:56.2525884Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-09-27T15:30:56.2526297Z [0., 0.]])} 2022-09-27T15:30:56.2531021Z 2022-09-27 15:30:56,252 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-09-27T15:30:56.2531611Z [ 1., -1.], 2022-09-27T15:30:56.2531866Z [ 1., 1.], 2022-09-27T15:30:56.2532552Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:56.2533461Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:56.2533951Z tensor([[-1., 1.], 2022-09-27T15:30:56.2534304Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-09-27T15:30:56.2534585Z [0., 0.]])} 2022-09-27T15:30:56.2610282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:56.2628585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:56.2629527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:56.2636432Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:30:56.2789282Z 2022-09-27 15:30:56,277 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-09-27T15:30:56.2790223Z [-1., -1.], 2022-09-27T15:30:56.2790796Z [-1., 1.], 2022-09-27T15:30:56.2791860Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:56.2793130Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:56.2793887Z tensor([[-1., 1.], 2022-09-27T15:30:56.2794329Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-09-27T15:30:56.2795028Z [0., 0.]])} 2022-09-27T15:30:56.2796298Z 2022-09-27 15:30:56,278 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-09-27T15:30:56.2797039Z [ 1., -1.], 2022-09-27T15:30:56.2797307Z [ 1., 1.], 2022-09-27T15:30:56.2798054Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:56.2798827Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:56.2799477Z tensor([[-1., 1.], 2022-09-27T15:30:56.2800489Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-09-27T15:30:56.2801093Z [0., 0.]])} 2022-09-27T15:30:56.2802190Z 2022-09-27 15:30:56,278 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:30:56.2803226Z [ 1., 1.], 2022-09-27T15:30:56.2803648Z [ 1., -1.], 2022-09-27T15:30:56.2804801Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:56.2805842Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:56.2806567Z tensor([[-1., 1.], 2022-09-27T15:30:56.2807171Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-09-27T15:30:56.2807582Z [0., 0.]])} 2022-09-27T15:30:56.2809091Z 2022-09-27 15:30:56,280 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:30:56.2809970Z [-1., 1.], 2022-09-27T15:30:56.2810546Z [-1., -1.], 2022-09-27T15:30:56.2811514Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:56.2812711Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:56.2813440Z tensor([[-1., 1.], 2022-09-27T15:30:56.2813863Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-09-27T15:30:56.2814441Z [0., 0.]])} 2022-09-27T15:30:56.3026042Z 2022-09-27 15:30:56,301 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-09-27T15:30:56.3026988Z [-1., -1.], 2022-09-27T15:30:56.3027363Z [-1., 1.], 2022-09-27T15:30:56.3028226Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:56.3029024Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:56.3029776Z tensor([[-1., 1.], 2022-09-27T15:30:56.3030386Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-09-27T15:30:56.3030960Z [0., 0.]])} 2022-09-27T15:30:56.3032106Z 2022-09-27 15:30:56,301 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:30:56.3032949Z [-1., 1.], 2022-09-27T15:30:56.3033523Z [-1., -1.], 2022-09-27T15:30:56.3034752Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:56.3035670Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:56.3036110Z tensor([[-1., 1.], 2022-09-27T15:30:56.3036609Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-09-27T15:30:56.3036866Z [0., 0.]])} 2022-09-27T15:30:56.3046679Z 2022-09-27 15:30:56,304 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-09-27T15:30:56.3047596Z [ 1., -1.], 2022-09-27T15:30:56.3048095Z [ 1., 1.], 2022-09-27T15:30:56.3049093Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:56.3050070Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:56.3050747Z tensor([[-1., 1.], 2022-09-27T15:30:56.3051318Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-09-27T15:30:56.3051669Z [0., 0.]])} 2022-09-27T15:30:56.3053410Z 2022-09-27 15:30:56,304 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:30:56.3054202Z [ 1., 1.], 2022-09-27T15:30:56.3054601Z [ 1., -1.], 2022-09-27T15:30:56.3055877Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:30:56.3056954Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-09-27T15:30:56.3057667Z tensor([[-1., 1.], 2022-09-27T15:30:56.3058266Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-09-27T15:30:56.3058831Z [0., 0.]])} 2022-09-27T15:30:56.3121239Z 2022-09-27 15:30:56,311 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0... 2022-09-27T15:30:56.3126735Z 2022-09-27 15:30:56,312 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1... 2022-09-27T15:30:56.3134579Z 2022-09-27 15:30:56,313 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2... 2022-09-27T15:30:56.3140809Z 2022-09-27 15:30:56,313 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3... 2022-09-27T15:30:56.3147755Z 2022-09-27 15:30:56,314 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker. 2022-09-27T15:30:56.9124438Z ok (3.861s) 2022-09-27T15:30:56.9126208Z 2022-09-27T15:30:56.9126664Z ---------------------------------------------------------------------- 2022-09-27T15:30:56.9127095Z Ran 1 test in 3.862s 2022-09-27T15:30:56.9127304Z 2022-09-27T15:30:56.9127394Z OK 2022-09-27T15:30:56.9127489Z 2022-09-27T15:30:56.9127588Z Generating XML reports... 2022-09-27T15:30:56.9161306Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220927153053.xml 2022-09-27T15:30:58.2743541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:30:58.2743944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:30:58.2751369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:30:58.2751812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:30:58.4370184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwn2un7qs 2022-09-27T15:30:58.4371832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwn2un7qs/_remote_module_non_scriptable.py 2022-09-27T15:30:58.6969718Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:30:58.6979095Z 2022-09-27T15:30:58.6979197Z Running tests... 2022-09-27T15:30:58.6979644Z ---------------------------------------------------------------------- 2022-09-27T15:30:58.9426182Z test_backward_ddp_outside_uneven_inputs (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4825 2022-09-27T15:30:58.9452175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4826 2022-09-27T15:30:58.9478790Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4827 2022-09-27T15:30:58.9506834Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4828 2022-09-27T15:30:58.9534519Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 4829 2022-09-27T15:30:58.9564849Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 4830 2022-09-27T15:31:00.2259450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:00.2260126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:00.2620395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:00.2621375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:00.3867632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:00.3868581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:00.3892723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporq0a4d2 2022-09-27T15:31:00.3895629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporq0a4d2/_remote_module_non_scriptable.py 2022-09-27T15:31:00.4214124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:00.4214718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:00.4817951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:00.4818596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:00.5311115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:00.5311893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:00.5477295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7tdj04vl 2022-09-27T15:31:00.5478422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7tdj04vl/_remote_module_non_scriptable.py 2022-09-27T15:31:00.6395977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:00.6396583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:00.6500829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:31:00.6800904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:00.6801558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:00.6916628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:00.6917253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:00.7289493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29a5wiki 2022-09-27T15:31:00.7290293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29a5wiki/_remote_module_non_scriptable.py 2022-09-27T15:31:00.7291734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:00.7292371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:00.7307827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:00.7308540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:00.7659433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:00.7660205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:00.8093962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:31:00.8945069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcnpo0go 2022-09-27T15:31:00.8945795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcnpo0go/_remote_module_non_scriptable.py 2022-09-27T15:31:00.8950148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5qpj5uze 2022-09-27T15:31:00.8952090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5qpj5uze/_remote_module_non_scriptable.py 2022-09-27T15:31:00.9360852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2vvs98k 2022-09-27T15:31:00.9362209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2vvs98k/_remote_module_non_scriptable.py 2022-09-27T15:31:00.9905091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-09-27T15:31:01.1565540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:31:01.1995957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:31:01.2264039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-09-27T15:31:01.5508418Z 2022-09-27 15:31:01,549 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0... 2022-09-27T15:31:01.5511256Z 2022-09-27 15:31:01,550 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3] 2022-09-27T15:31:01.5606084Z 2022-09-27 15:31:01,559 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1... 2022-09-27T15:31:01.5607272Z 2022-09-27 15:31:01,560 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3] 2022-09-27T15:31:01.5707184Z 2022-09-27 15:31:01,569 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3... 2022-09-27T15:31:01.5712123Z 2022-09-27 15:31:01,569 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2... 2022-09-27T15:31:01.5718667Z 2022-09-27 15:31:01,570 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3] 2022-09-27T15:31:01.5723658Z 2022-09-27 15:31:01,571 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3] 2022-09-27T15:31:01.5803642Z 2022-09-27 15:31:01,579 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process... 2022-09-27T15:31:01.5814531Z 2022-09-27 15:31:01,580 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running. 2022-09-27T15:31:01.6143170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:31:01.6147914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:31:01.6152423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:31:01.6243684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:31:01.6246239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-09-27T15:31:01.6258107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-09-27T15:31:01.6266341Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:31:01.6267511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:31:01.6269006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:31:01.6270143Z 2022-09-27 15:31:01,625 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0... 2022-09-27T15:31:01.6276250Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:31:01.6277127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-09-27T15:31:01.6349598Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:31:01.6354060Z 2022-09-27 15:31:01,634 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3... 2022-09-27T15:31:01.6357161Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:31:01.6363007Z 2022-09-27 15:31:01,635 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master 2022-09-27T15:31:01.6365777Z 2022-09-27 15:31:01,635 ddp_under_dist_autograd_test.py:396 INFO p:process 5 t:MainThread: Running DDP + RPC test with simulating uneven inputs across trainers. 2022-09-27T15:31:01.6379493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-09-27T15:31:01.6380688Z 2022-09-27 15:31:01,636 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1... 2022-09-27T15:31:01.6381761Z 2022-09-27 15:31:01,637 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2... 2022-09-27T15:31:01.6595381Z 2022-09-27 15:31:01,658 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3 2022-09-27T15:31:01.6598301Z 2022-09-27 15:31:01,659 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3 2022-09-27T15:31:01.8079108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-09-27T15:31:01.8088134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-09-27T15:31:01.8093652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-09-27T15:31:01.8103157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-09-27T15:31:01.8104478Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:31:01.8116658Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:31:01.8184268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:31:01.8202664Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:31:01.8204120Z 2022-09-27 15:31:01,818 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:31:01.8205340Z 2022-09-27 15:31:01,819 ddp_under_dist_autograd_test.py:202 INFO p:process 0 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-09-27T15:31:01.8206765Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:31:01.8208263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-09-27T15:31:01.8211415Z 2022-09-27 15:31:01,820 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:31:01.8212821Z 2022-09-27 15:31:01,820 ddp_under_dist_autograd_test.py:202 INFO p:process 1 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-09-27T15:31:01.8257460Z 2022-09-27 15:31:01,825 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:31:01.8258599Z 2022-09-27 15:31:01,825 ddp_under_dist_autograd_test.py:202 INFO p:process 2 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-09-27T15:31:01.8267245Z 2022-09-27 15:31:01,826 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:31:01.8271254Z 2022-09-27 15:31:01,826 ddp_under_dist_autograd_test.py:202 INFO p:process 3 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-09-27T15:31:01.8450277Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-09-27T15:31:01.8454051Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-09-27T15:31:01.8467302Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-09-27T15:31:01.8468268Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-09-27T15:31:01.8646699Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-09-27T15:31:01.8647551Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-09-27T15:31:01.8676524Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:630: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-09-27T15:31:01.8677823Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-09-27T15:31:01.8704757Z 2022-09-27 15:31:01,869 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-09-27T15:31:01.8712157Z 2022-09-27 15:31:01,870 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-09-27T15:31:01.8713315Z 2022-09-27 15:31:01,870 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-09-27T15:31:01.8719373Z 2022-09-27 15:31:01,871 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-3: Trainer reduced input patches from 2 2022-09-27T15:31:01.8720205Z to 1 to simulate uneven inputs. 2022-09-27T15:31:01.8724282Z 2022-09-27 15:31:01,872 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-09-27T15:31:01.8731422Z 2022-09-27 15:31:01,872 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-3: Trainer reduced input patches from 2 2022-09-27T15:31:01.8732135Z to 1 to simulate uneven inputs. 2022-09-27T15:31:01.9530515Z 2022-09-27 15:31:01,952 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-09-27T15:31:01.9533229Z [ 1., -1.], 2022-09-27T15:31:01.9533720Z [ 1., 1.], 2022-09-27T15:31:01.9540695Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:01.9541735Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-09-27T15:31:01.9542179Z tensor([[-1., 1.], 2022-09-27T15:31:01.9542672Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-09-27T15:31:01.9542956Z [ 0., 4.]])} 2022-09-27T15:31:01.9543727Z 2022-09-27 15:31:01,952 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:31:01.9544328Z [ 1., 1.], 2022-09-27T15:31:01.9544608Z [ 1., -1.], 2022-09-27T15:31:01.9545307Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:01.9546039Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-09-27T15:31:01.9546447Z tensor([[-1., 1.], 2022-09-27T15:31:01.9546765Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-09-27T15:31:01.9547295Z [ 0., 4.]])} 2022-09-27T15:31:01.9548467Z 2022-09-27 15:31:01,953 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:31:01.9549078Z [-1., 1.], 2022-09-27T15:31:01.9552867Z [-1., -1.], 2022-09-27T15:31:01.9559794Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:01.9560503Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-09-27T15:31:01.9560964Z tensor([[-1., 1.], 2022-09-27T15:31:01.9561425Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-09-27T15:31:01.9561822Z [ 0., 4.]])} 2022-09-27T15:31:01.9562758Z 2022-09-27 15:31:01,954 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-09-27T15:31:01.9563450Z [-1., -1.], 2022-09-27T15:31:01.9563834Z [-1., 1.], 2022-09-27T15:31:01.9564675Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:01.9565573Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-09-27T15:31:01.9566122Z tensor([[-1., 1.], 2022-09-27T15:31:01.9566548Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-09-27T15:31:01.9566958Z [ 0., 4.]])} 2022-09-27T15:31:01.9575262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:31:01.9575787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-09-27T15:31:01.9701294Z 2022-09-27 15:31:01,969 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:31:01.9701862Z [ 1., 1.], 2022-09-27T15:31:01.9702433Z [ 1., -1.], 2022-09-27T15:31:01.9703128Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:01.9703791Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-09-27T15:31:01.9704063Z tensor([[-1., 1.], 2022-09-27T15:31:01.9704322Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-09-27T15:31:01.9705117Z [-2., 2.]])} 2022-09-27T15:31:01.9716506Z 2022-09-27 15:31:01,971 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:31:01.9717232Z [-1., 1.], 2022-09-27T15:31:01.9717675Z [-1., -1.], 2022-09-27T15:31:01.9718507Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:01.9719399Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-09-27T15:31:01.9719968Z tensor([[-1., 1.], 2022-09-27T15:31:01.9720567Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-09-27T15:31:01.9721073Z [-2., 2.]])} 2022-09-27T15:31:01.9757120Z 2022-09-27 15:31:01,975 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-4: Trainer reduced input patches from 2 2022-09-27T15:31:01.9757587Z to 1 to simulate uneven inputs. 2022-09-27T15:31:01.9758204Z 2022-09-27 15:31:01,975 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-4: Trainer reduced input patches from 2 2022-09-27T15:31:01.9758612Z to 1 to simulate uneven inputs. 2022-09-27T15:31:01.9922129Z 2022-09-27 15:31:01,991 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:31:01.9923338Z [ 1., 1.], 2022-09-27T15:31:01.9924053Z [ 1., -1.], 2022-09-27T15:31:01.9924711Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:01.9926177Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-09-27T15:31:01.9926584Z tensor([[-1., 1.], 2022-09-27T15:31:01.9928214Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-09-27T15:31:01.9928514Z [ 0., 4.]])} 2022-09-27T15:31:01.9936955Z 2022-09-27 15:31:01,991 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-09-27T15:31:01.9937431Z [-1., -1.], 2022-09-27T15:31:01.9937617Z [-1., 1.], 2022-09-27T15:31:01.9938045Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:01.9938473Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-09-27T15:31:01.9938738Z tensor([[-1., 1.], 2022-09-27T15:31:01.9938947Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-09-27T15:31:01.9939144Z [ 0., 4.]])} 2022-09-27T15:31:01.9939572Z 2022-09-27 15:31:01,991 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-09-27T15:31:01.9939878Z [ 1., -1.], 2022-09-27T15:31:01.9940039Z [ 1., 1.], 2022-09-27T15:31:01.9940428Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:01.9940846Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-09-27T15:31:01.9941097Z tensor([[-1., 1.], 2022-09-27T15:31:01.9941308Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-09-27T15:31:01.9941509Z [ 0., 4.]])} 2022-09-27T15:31:01.9942113Z 2022-09-27 15:31:01,991 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:31:01.9942632Z [-1., 1.], 2022-09-27T15:31:01.9943011Z [-1., -1.], 2022-09-27T15:31:01.9943626Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:01.9944372Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-09-27T15:31:01.9944838Z tensor([[-1., 1.], 2022-09-27T15:31:01.9945220Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-09-27T15:31:01.9945496Z [ 0., 4.]])} 2022-09-27T15:31:02.0034007Z 2022-09-27 15:31:02,002 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:31:02.0034414Z [-1., 1.], 2022-09-27T15:31:02.0034932Z [-1., -1.], 2022-09-27T15:31:02.0035780Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:02.0036682Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-09-27T15:31:02.0037229Z tensor([[-1., 1.], 2022-09-27T15:31:02.0037778Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-09-27T15:31:02.0038261Z [-2., 2.]])} 2022-09-27T15:31:02.0061865Z 2022-09-27 15:31:02,005 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:31:02.0062671Z [ 1., 1.], 2022-09-27T15:31:02.0063101Z [ 1., -1.], 2022-09-27T15:31:02.0064251Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:02.0065764Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-09-27T15:31:02.0066575Z tensor([[-1., 1.], 2022-09-27T15:31:02.0067165Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-09-27T15:31:02.0067811Z [-2., 2.]])} 2022-09-27T15:31:02.0097607Z 2022-09-27 15:31:02,009 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-5: Trainer reduced input patches from 2 2022-09-27T15:31:02.0099304Z to 1 to simulate uneven inputs. 2022-09-27T15:31:02.0100186Z 2022-09-27 15:31:02,009 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-5: Trainer reduced input patches from 2 2022-09-27T15:31:02.0100815Z to 1 to simulate uneven inputs. 2022-09-27T15:31:02.0247192Z 2022-09-27 15:31:02,024 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:31:02.0248237Z [-1., 1.], 2022-09-27T15:31:02.0251918Z [-1., -1.], 2022-09-27T15:31:02.0252831Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:02.0253744Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-09-27T15:31:02.0254298Z tensor([[-1., 1.], 2022-09-27T15:31:02.0254750Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-09-27T15:31:02.0255182Z [ 0., 4.]])} 2022-09-27T15:31:02.0276142Z 2022-09-27 15:31:02,025 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-09-27T15:31:02.0276743Z [-1., -1.], 2022-09-27T15:31:02.0277149Z [-1., 1.], 2022-09-27T15:31:02.0278002Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:02.0278908Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-09-27T15:31:02.0279452Z tensor([[-1., 1.], 2022-09-27T15:31:02.0279895Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-09-27T15:31:02.0280319Z [ 0., 4.]])} 2022-09-27T15:31:02.0317791Z 2022-09-27 15:31:02,030 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:31:02.0318492Z [ 1., 1.], 2022-09-27T15:31:02.0318917Z [ 1., -1.], 2022-09-27T15:31:02.0319760Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:02.0320621Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-09-27T15:31:02.0321176Z tensor([[-1., 1.], 2022-09-27T15:31:02.0321612Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-09-27T15:31:02.0322027Z [ 0., 4.]])} 2022-09-27T15:31:02.0322966Z 2022-09-27 15:31:02,030 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-09-27T15:31:02.0323673Z [ 1., -1.], 2022-09-27T15:31:02.0323997Z [ 1., 1.], 2022-09-27T15:31:02.0324831Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:02.0325727Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-09-27T15:31:02.0326286Z tensor([[-1., 1.], 2022-09-27T15:31:02.0326712Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-09-27T15:31:02.0327125Z [ 0., 4.]])} 2022-09-27T15:31:02.0502878Z 2022-09-27 15:31:02,049 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:31:02.0503901Z [ 1., 1.], 2022-09-27T15:31:02.0504297Z [ 1., -1.], 2022-09-27T15:31:02.0505134Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:02.0506410Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-09-27T15:31:02.0506944Z tensor([[-1., 1.], 2022-09-27T15:31:02.0507519Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-09-27T15:31:02.0509342Z [-2., 2.]])} 2022-09-27T15:31:02.0510793Z 2022-09-27 15:31:02,050 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:31:02.0511684Z [-1., 1.], 2022-09-27T15:31:02.0512452Z [-1., -1.], 2022-09-27T15:31:02.0513618Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:02.0514677Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-09-27T15:31:02.0515237Z tensor([[-1., 1.], 2022-09-27T15:31:02.0516043Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-09-27T15:31:02.0516531Z [-2., 2.]])} 2022-09-27T15:31:02.0575138Z 2022-09-27 15:31:02,057 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0... 2022-09-27T15:31:02.0582917Z 2022-09-27 15:31:02,057 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1... 2022-09-27T15:31:02.0589443Z 2022-09-27 15:31:02,058 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2... 2022-09-27T15:31:02.0596102Z 2022-09-27 15:31:02,059 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3... 2022-09-27T15:31:02.0603665Z 2022-09-27 15:31:02,059 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker. 2022-09-27T15:31:02.1425170Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-09-27T15:31:02.1426094Z _warnings.warn(warn_message, ResourceWarning) 2022-09-27T15:31:02.1498619Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-09-27T15:31:02.1499338Z _warnings.warn(warn_message, ResourceWarning) 2022-09-27T15:31:02.1500173Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-09-27T15:31:02.1500828Z _warnings.warn(warn_message, ResourceWarning) 2022-09-27T15:31:02.1501670Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-09-27T15:31:02.1502309Z _warnings.warn(warn_message, ResourceWarning) 2022-09-27T15:31:02.6654370Z ok (3.967s) 2022-09-27T15:31:02.6654531Z 2022-09-27T15:31:02.6657256Z ---------------------------------------------------------------------- 2022-09-27T15:31:02.6657559Z Ran 1 test in 3.967s 2022-09-27T15:31:02.6657683Z 2022-09-27T15:31:02.6657743Z OK 2022-09-27T15:31:02.6657834Z 2022-09-27T15:31:02.6657926Z Generating XML reports... 2022-09-27T15:31:02.6693387Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220927153058.xml 2022-09-27T15:31:04.0480219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:04.0480610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:04.0489809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:04.0490316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:04.2125791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90j7_5nh 2022-09-27T15:31:04.2126673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90j7_5nh/_remote_module_non_scriptable.py 2022-09-27T15:31:04.4745547Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:04.4755336Z 2022-09-27T15:31:04.4755481Z Running tests... 2022-09-27T15:31:04.4756061Z ---------------------------------------------------------------------- 2022-09-27T15:31:04.7187422Z test_backward_no_ddp (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5248 2022-09-27T15:31:04.7214594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5249 2022-09-27T15:31:04.7242076Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5250 2022-09-27T15:31:04.7270897Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5251 2022-09-27T15:31:04.7310315Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 5252 2022-09-27T15:31:04.7331441Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 5253 2022-09-27T15:31:06.0575981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:06.0577085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:06.1072244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:06.1072908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:06.1594067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:06.1595022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:06.1981708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:06.1982401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:06.2360609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppf1c1cn_ 2022-09-27T15:31:06.2361713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppf1c1cn_/_remote_module_non_scriptable.py 2022-09-27T15:31:06.2836303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:06.2837175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:06.3262647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptay2qm0t 2022-09-27T15:31:06.3263992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptay2qm0t/_remote_module_non_scriptable.py 2022-09-27T15:31:06.3347918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:06.3348595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:06.4122553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:06.4123236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:06.4478042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:06.4478637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:06.4644767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:06.4645420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:06.4990168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:06.4991038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:06.4995054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:31:06.5366272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmur_u1om 2022-09-27T15:31:06.5367244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:06.5367783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:06.5370062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmur_u1om/_remote_module_non_scriptable.py 2022-09-27T15:31:06.5884399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:06.5885000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:06.5913950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-09-27T15:31:06.6482038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt33w2l2k 2022-09-27T15:31:06.6482567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt33w2l2k/_remote_module_non_scriptable.py 2022-09-27T15:31:06.6995226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkwwvbfnv 2022-09-27T15:31:06.6996446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkwwvbfnv/_remote_module_non_scriptable.py 2022-09-27T15:31:06.7652491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rxehv2r 2022-09-27T15:31:06.7653238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rxehv2r/_remote_module_non_scriptable.py 2022-09-27T15:31:06.8262703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:31:06.9016051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:31:06.9649702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-09-27T15:31:07.0219585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:31:07.3287030Z 2022-09-27 15:31:07,327 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0... 2022-09-27T15:31:07.3293915Z 2022-09-27 15:31:07,328 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3] 2022-09-27T15:31:07.3305311Z 2022-09-27 15:31:07,329 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1... 2022-09-27T15:31:07.3310392Z 2022-09-27 15:31:07,330 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3] 2022-09-27T15:31:07.3393373Z 2022-09-27 15:31:07,338 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2... 2022-09-27T15:31:07.3394682Z 2022-09-27 15:31:07,338 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3] 2022-09-27T15:31:07.3586653Z 2022-09-27 15:31:07,358 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running. 2022-09-27T15:31:07.3588134Z 2022-09-27 15:31:07,358 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3... 2022-09-27T15:31:07.3591109Z 2022-09-27 15:31:07,358 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3] 2022-09-27T15:31:07.3606506Z 2022-09-27 15:31:07,360 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process... 2022-09-27T15:31:07.3719353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:31:07.3824064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:31:07.3831869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:31:07.4026698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:31:07.4128733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-09-27T15:31:07.4129634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-09-27T15:31:07.4135985Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:31:07.4143151Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:31:07.4144311Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:31:07.4145454Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:31:07.4149662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:31:07.4155092Z 2022-09-27 15:31:07,414 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0... 2022-09-27T15:31:07.4156227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-09-27T15:31:07.4157355Z 2022-09-27 15:31:07,415 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1... 2022-09-27T15:31:07.4242865Z 2022-09-27 15:31:07,423 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2... 2022-09-27T15:31:07.4249549Z 2022-09-27 15:31:07,424 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3... 2022-09-27T15:31:07.4256504Z 2022-09-27 15:31:07,425 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master 2022-09-27T15:31:07.4352071Z 2022-09-27 15:31:07,434 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:31:07.4356049Z 2022-09-27 15:31:07,434 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params. 2022-09-27T15:31:07.4569501Z 2022-09-27 15:31:07,456 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3 2022-09-27T15:31:07.4577236Z 2022-09-27 15:31:07,456 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3 2022-09-27T15:31:07.4840967Z 2022-09-27 15:31:07,483 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-09-27T15:31:07.4841995Z [ 1., -1.], 2022-09-27T15:31:07.4844992Z [ 1., 1.], 2022-09-27T15:31:07.4846082Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:07.4847834Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing: 2022-09-27T15:31:07.4848628Z tensor([[-1., 1.], 2022-09-27T15:31:07.4849355Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-09-27T15:31:07.4850019Z [ 0., -16.]])} 2022-09-27T15:31:07.5388027Z 2022-09-27 15:31:07,538 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:31:07.5389582Z 2022-09-27 15:31:07,538 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params. 2022-09-27T15:31:07.5414444Z 2022-09-27 15:31:07,540 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:31:07.5416033Z 2022-09-27 15:31:07,541 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params. 2022-09-27T15:31:07.5643259Z 2022-09-27 15:31:07,563 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-09-27T15:31:07.5644003Z [-1., -1.], 2022-09-27T15:31:07.5644408Z [-1., 1.], 2022-09-27T15:31:07.5645235Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:07.5646165Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., -28., -28.]]), Parameter containing: 2022-09-27T15:31:07.5646733Z tensor([[-1., 1.], 2022-09-27T15:31:07.5647329Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-09-27T15:31:07.5647802Z [ 16., -16.]])} 2022-09-27T15:31:07.5652079Z 2022-09-27 15:31:07,564 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:31:07.5656271Z [-1., 1.], 2022-09-27T15:31:07.5656760Z [-1., -1.], 2022-09-27T15:31:07.5657622Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:07.5658537Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing: 2022-09-27T15:31:07.5659073Z tensor([[-1., 1.], 2022-09-27T15:31:07.5659651Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-09-27T15:31:07.5660145Z [-16., 16.]])} 2022-09-27T15:31:07.5733519Z 2022-09-27 15:31:07,572 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-09-27T15:31:07.5734743Z 2022-09-27 15:31:07,573 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params. 2022-09-27T15:31:07.5927594Z 2022-09-27 15:31:07,592 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:31:07.5928558Z [ 1., 1.], 2022-09-27T15:31:07.5929476Z [ 1., -1.], 2022-09-27T15:31:07.5930366Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:07.5931240Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing: 2022-09-27T15:31:07.5931766Z tensor([[-1., 1.], 2022-09-27T15:31:07.5932217Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-09-27T15:31:07.5932611Z [ 0., 16.]])} 2022-09-27T15:31:07.6097236Z 2022-09-27 15:31:07,609 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-09-27T15:31:07.6098295Z [-1., -1.], 2022-09-27T15:31:07.6099250Z [-1., 1.], 2022-09-27T15:31:07.6100073Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:07.6100691Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., -28., -28.]]), Parameter containing: 2022-09-27T15:31:07.6101091Z tensor([[-1., 1.], 2022-09-27T15:31:07.6101401Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-09-27T15:31:07.6101639Z [ 16., -16.]])} 2022-09-27T15:31:07.6155391Z 2022-09-27 15:31:07,615 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:31:07.6156237Z [ 1., 1.], 2022-09-27T15:31:07.6156630Z [ 1., -1.], 2022-09-27T15:31:07.6157399Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:07.6158485Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing: 2022-09-27T15:31:07.6158968Z tensor([[-1., 1.], 2022-09-27T15:31:07.6159450Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-09-27T15:31:07.6159754Z [ 0., 16.]])} 2022-09-27T15:31:07.6172781Z 2022-09-27 15:31:07,616 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-09-27T15:31:07.6173762Z [ 1., -1.], 2022-09-27T15:31:07.6174405Z [ 1., 1.], 2022-09-27T15:31:07.6174913Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:07.6175412Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing: 2022-09-27T15:31:07.6175802Z tensor([[-1., 1.], 2022-09-27T15:31:07.6176188Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-09-27T15:31:07.6176546Z [ 0., -16.]])} 2022-09-27T15:31:07.6181291Z 2022-09-27 15:31:07,617 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:31:07.6181818Z [-1., 1.], 2022-09-27T15:31:07.6182097Z [-1., -1.], 2022-09-27T15:31:07.6182704Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:07.6183398Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing: 2022-09-27T15:31:07.6183799Z tensor([[-1., 1.], 2022-09-27T15:31:07.6184229Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-09-27T15:31:07.6184603Z [-16., 16.]])} 2022-09-27T15:31:07.6318808Z 2022-09-27 15:31:07,631 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-09-27T15:31:07.6319395Z [ 1., -1.], 2022-09-27T15:31:07.6319635Z [ 1., 1.], 2022-09-27T15:31:07.6320235Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:07.6320801Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing: 2022-09-27T15:31:07.6321088Z tensor([[-1., 1.], 2022-09-27T15:31:07.6321365Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-09-27T15:31:07.6321638Z [ 0., -16.]])} 2022-09-27T15:31:07.6347804Z 2022-09-27 15:31:07,634 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-09-27T15:31:07.6348638Z [ 1., 1.], 2022-09-27T15:31:07.6349191Z [ 1., -1.], 2022-09-27T15:31:07.6350058Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:07.6350788Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing: 2022-09-27T15:31:07.6351235Z tensor([[-1., 1.], 2022-09-27T15:31:07.6351637Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-09-27T15:31:07.6351950Z [ 0., 16.]])} 2022-09-27T15:31:07.6392399Z 2022-09-27 15:31:07,638 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-09-27T15:31:07.6393100Z [-1., -1.], 2022-09-27T15:31:07.6393428Z [-1., 1.], 2022-09-27T15:31:07.6394458Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:07.6395466Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., -28., -28.]]), Parameter containing: 2022-09-27T15:31:07.6396076Z tensor([[-1., 1.], 2022-09-27T15:31:07.6396616Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-09-27T15:31:07.6397060Z [ 16., -16.]])} 2022-09-27T15:31:07.6461504Z 2022-09-27 15:31:07,645 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-09-27T15:31:07.6462097Z [-1., 1.], 2022-09-27T15:31:07.6462427Z [-1., -1.], 2022-09-27T15:31:07.6463035Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-09-27T15:31:07.6463678Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing: 2022-09-27T15:31:07.6464092Z tensor([[-1., 1.], 2022-09-27T15:31:07.6464524Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-09-27T15:31:07.6464883Z [-16., 16.]])} 2022-09-27T15:31:07.6511907Z 2022-09-27 15:31:07,650 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0... 2022-09-27T15:31:07.6519539Z 2022-09-27 15:31:07,651 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1... 2022-09-27T15:31:07.6527671Z 2022-09-27 15:31:07,652 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2... 2022-09-27T15:31:07.6536418Z 2022-09-27 15:31:07,653 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3... 2022-09-27T15:31:07.6543801Z 2022-09-27 15:31:07,654 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker. 2022-09-27T15:31:08.2417502Z ok (3.766s) 2022-09-27T15:31:08.2417669Z 2022-09-27T15:31:08.2420516Z ---------------------------------------------------------------------- 2022-09-27T15:31:08.2420997Z Ran 1 test in 3.766s 2022-09-27T15:31:08.2421111Z 2022-09-27T15:31:08.2421180Z OK 2022-09-27T15:31:08.2421279Z 2022-09-27T15:31:08.2421372Z Generating XML reports... 2022-09-27T15:31:08.2456461Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220927153104.xml 2022-09-27T15:31:09.6117814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:09.6118199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:09.6125780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:09.6126271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:09.7742770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_oqwrqr 2022-09-27T15:31:09.7744035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_oqwrqr/_remote_module_non_scriptable.py 2022-09-27T15:31:10.0417648Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:10.0427671Z 2022-09-27T15:31:10.0427811Z Running tests... 2022-09-27T15:31:10.0428412Z ---------------------------------------------------------------------- 2022-09-27T15:31:10.0435200Z test_async_dist_autograd (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:31:10.2894863Z This test ensures async processing for distributed autograd works ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5653 2022-09-27T15:31:10.2921472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5654 2022-09-27T15:31:10.2949290Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5655 2022-09-27T15:31:10.2984085Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5656 2022-09-27T15:31:11.3966427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:11.3967182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:11.4238710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:11.4239323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:11.4363137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:11.4363806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:11.4615058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:11.4615691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:11.4637131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:11.4637807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:11.4709826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:11.4710335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:11.5016680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:11.5017369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:11.5103044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:11.5103725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:11.5653565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwogdfbi 2022-09-27T15:31:11.5654683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwogdfbi/_remote_module_non_scriptable.py 2022-09-27T15:31:11.5915582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj0wnmzi_ 2022-09-27T15:31:11.5916548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj0wnmzi_/_remote_module_non_scriptable.py 2022-09-27T15:31:11.6299573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2dcnuq4g 2022-09-27T15:31:11.6300434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2dcnuq4g/_remote_module_non_scriptable.py 2022-09-27T15:31:11.6368361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xtgd2go 2022-09-27T15:31:11.6369496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xtgd2go/_remote_module_non_scriptable.py 2022-09-27T15:31:11.8261394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:31:11.8503763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:31:11.8906595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:31:11.8964002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:31:12.1490512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:31:12.1590420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:31:12.1704322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:31:12.1705578Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:31:12.1706999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:31:12.1708140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:31:12.1709096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:31:12.1710131Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:31:12.7044234Z ok (2.661s) 2022-09-27T15:31:12.7044517Z 2022-09-27T15:31:12.7045019Z ---------------------------------------------------------------------- 2022-09-27T15:31:12.7045441Z Ran 1 test in 2.662s 2022-09-27T15:31:12.7045650Z 2022-09-27T15:31:12.7045757Z OK 2022-09-27T15:31:12.7045920Z 2022-09-27T15:31:12.7046086Z Generating XML reports... 2022-09-27T15:31:12.7080939Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153110.xml 2022-09-27T15:31:14.0896765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:14.0897198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:14.0904565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:14.0904975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:14.2472601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpycmu3395 2022-09-27T15:31:14.2473505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpycmu3395/_remote_module_non_scriptable.py 2022-09-27T15:31:14.5093771Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:14.5103558Z 2022-09-27T15:31:14.5103699Z Running tests... 2022-09-27T15:31:14.5104161Z ---------------------------------------------------------------------- 2022-09-27T15:31:14.7534743Z test_autograd_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5968 2022-09-27T15:31:14.7561100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5969 2022-09-27T15:31:14.7587593Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5970 2022-09-27T15:31:14.7615481Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5971 2022-09-27T15:31:15.8383467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:15.8384048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:15.8738226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:15.8738961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:15.8822530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:15.8823112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:15.8840529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:15.8841208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:15.9137702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:15.9138329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:15.9203958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:15.9204627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:15.9225143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:15.9225842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:15.9517045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:15.9517749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:16.0010523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2sb0d0x1 2022-09-27T15:31:16.0011576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2sb0d0x1/_remote_module_non_scriptable.py 2022-09-27T15:31:16.0471544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvb2ojqk 2022-09-27T15:31:16.0472318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvb2ojqk/_remote_module_non_scriptable.py 2022-09-27T15:31:16.0492874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8see0my 2022-09-27T15:31:16.0494061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8see0my/_remote_module_non_scriptable.py 2022-09-27T15:31:16.0782618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptzekrvsi 2022-09-27T15:31:16.0783609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptzekrvsi/_remote_module_non_scriptable.py 2022-09-27T15:31:16.2566611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:31:16.3017926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:31:16.3053102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:31:16.3356426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:31:17.5685423Z ok (3.058s) 2022-09-27T15:31:17.5685642Z 2022-09-27T15:31:17.5686008Z ---------------------------------------------------------------------- 2022-09-27T15:31:17.5686267Z Ran 1 test in 3.058s 2022-09-27T15:31:17.5686385Z 2022-09-27T15:31:17.5686460Z OK 2022-09-27T15:31:17.5686606Z 2022-09-27T15:31:17.5686709Z Generating XML reports... 2022-09-27T15:31:17.5721968Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153114.xml 2022-09-27T15:31:18.9691534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:18.9691916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:18.9701345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:18.9701756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:19.1298334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3qts42xd 2022-09-27T15:31:19.1299567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3qts42xd/_remote_module_non_scriptable.py 2022-09-27T15:31:19.3931907Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:19.3941815Z 2022-09-27T15:31:19.3941927Z Running tests... 2022-09-27T15:31:19.3942479Z ---------------------------------------------------------------------- 2022-09-27T15:31:19.6531218Z test_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6191 2022-09-27T15:31:19.6558248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6192 2022-09-27T15:31:19.6585610Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6193 2022-09-27T15:31:19.6613719Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6194 2022-09-27T15:31:20.7332672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:20.7333279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:20.7573611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:20.7574581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:20.7732892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:20.7733501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:20.7788627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:20.7789320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:20.7975828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:20.7976374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:20.8177299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:20.8178001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:20.8178802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:20.8179420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:20.8557557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:20.8558278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:20.9012638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgfsjgtgc 2022-09-27T15:31:20.9013679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgfsjgtgc/_remote_module_non_scriptable.py 2022-09-27T15:31:20.9259476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33589prq 2022-09-27T15:31:20.9260336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33589prq/_remote_module_non_scriptable.py 2022-09-27T15:31:20.9450165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpse1rmfvj 2022-09-27T15:31:20.9450992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpse1rmfvj/_remote_module_non_scriptable.py 2022-09-27T15:31:20.9816248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa40yv80o 2022-09-27T15:31:20.9816987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa40yv80o/_remote_module_non_scriptable.py 2022-09-27T15:31:21.1579621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:31:21.1839305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:31:21.2026350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:31:21.2385331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:31:21.9672142Z ok (2.573s) 2022-09-27T15:31:21.9672387Z 2022-09-27T15:31:21.9672907Z ---------------------------------------------------------------------- 2022-09-27T15:31:21.9673283Z Ran 1 test in 2.573s 2022-09-27T15:31:21.9673402Z 2022-09-27T15:31:21.9673449Z OK 2022-09-27T15:31:21.9673540Z 2022-09-27T15:31:21.9673637Z Generating XML reports... 2022-09-27T15:31:21.9709192Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153119.xml 2022-09-27T15:31:23.3517244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:23.3517916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:23.3527061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:23.3527782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:23.5107488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_54hkr6r 2022-09-27T15:31:23.5108388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_54hkr6r/_remote_module_non_scriptable.py 2022-09-27T15:31:23.7713788Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:23.7724419Z 2022-09-27T15:31:23.7724924Z Running tests... 2022-09-27T15:31:23.7725352Z ---------------------------------------------------------------------- 2022-09-27T15:31:24.0152394Z test_backward_autograd_engine_error (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6434 2022-09-27T15:31:24.0179210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6435 2022-09-27T15:31:24.0205556Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6436 2022-09-27T15:31:24.0233423Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6437 2022-09-27T15:31:25.1100311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:25.1100972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:25.1478830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:25.1479478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:25.1480668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:25.1481276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:25.1568233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:25.1568887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:25.1778313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:25.1779015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:25.1854092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:25.1854626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:25.1947823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:25.1948388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:25.2156864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:25.2157472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:25.2762424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09jlj1wv 2022-09-27T15:31:25.2763183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09jlj1wv/_remote_module_non_scriptable.py 2022-09-27T15:31:25.3121863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgfc_pom 2022-09-27T15:31:25.3122615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgfc_pom/_remote_module_non_scriptable.py 2022-09-27T15:31:25.3210575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7s3ubkzc 2022-09-27T15:31:25.3211586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7s3ubkzc/_remote_module_non_scriptable.py 2022-09-27T15:31:25.3429335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptr8z0tlr 2022-09-27T15:31:25.3430035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptr8z0tlr/_remote_module_non_scriptable.py 2022-09-27T15:31:25.5331029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:31:25.5691795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:31:25.5828813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:31:25.5994001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:31:26.3291803Z ok (2.556s) 2022-09-27T15:31:26.3292021Z 2022-09-27T15:31:26.3292308Z ---------------------------------------------------------------------- 2022-09-27T15:31:26.3292559Z Ran 1 test in 2.557s 2022-09-27T15:31:26.3292674Z 2022-09-27T15:31:26.3292736Z OK 2022-09-27T15:31:26.3292829Z 2022-09-27T15:31:26.3292908Z Generating XML reports... 2022-09-27T15:31:26.3328415Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153123.xml 2022-09-27T15:31:27.6964077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:27.6964473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:27.6972860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:27.6973276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:27.8652954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7kkuk2lr 2022-09-27T15:31:27.8653869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7kkuk2lr/_remote_module_non_scriptable.py 2022-09-27T15:31:28.1255479Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:28.1265384Z 2022-09-27T15:31:28.1265531Z Running tests... 2022-09-27T15:31:28.1265938Z ---------------------------------------------------------------------- 2022-09-27T15:31:28.3696023Z test_backward_complex_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6677 2022-09-27T15:31:28.3723771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6678 2022-09-27T15:31:28.3750939Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6679 2022-09-27T15:31:28.3778824Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6680 2022-09-27T15:31:29.4444820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:29.4445511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:29.4751668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:29.4752292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:29.4824983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:29.4825877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:29.4929488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:29.4930077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:29.4930794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:29.4931353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:29.5134722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:29.5135652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:29.5303233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:29.5304196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:29.5307292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:29.5307914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:29.6098057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0p2gb76 2022-09-27T15:31:29.6098980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0p2gb76/_remote_module_non_scriptable.py 2022-09-27T15:31:29.6411036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfovbfhhd 2022-09-27T15:31:29.6411755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfovbfhhd/_remote_module_non_scriptable.py 2022-09-27T15:31:29.6574466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmh9amq0v 2022-09-27T15:31:29.6575270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoi88hsn2 2022-09-27T15:31:29.6576223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmh9amq0v/_remote_module_non_scriptable.py 2022-09-27T15:31:29.6576993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoi88hsn2/_remote_module_non_scriptable.py 2022-09-27T15:31:29.8664867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:31:29.8989696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:31:29.9119260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:31:29.9141363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:31:30.5834455Z ok (2.457s) 2022-09-27T15:31:30.5834836Z 2022-09-27T15:31:30.5835345Z ---------------------------------------------------------------------- 2022-09-27T15:31:30.5835783Z Ran 1 test in 2.457s 2022-09-27T15:31:30.5835933Z 2022-09-27T15:31:30.5835995Z OK 2022-09-27T15:31:30.5836086Z 2022-09-27T15:31:30.5836181Z Generating XML reports... 2022-09-27T15:31:30.5872826Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153128.xml 2022-09-27T15:31:31.9684637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:31.9685031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:31.9692915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:31.9693284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:32.1283792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58mfqrra 2022-09-27T15:31:32.1284694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58mfqrra/_remote_module_non_scriptable.py 2022-09-27T15:31:32.3926219Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:32.3935358Z 2022-09-27T15:31:32.3935631Z Running tests... 2022-09-27T15:31:32.3936108Z ---------------------------------------------------------------------- 2022-09-27T15:31:32.6377659Z test_backward_different_dtypes (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6920 2022-09-27T15:31:32.6404438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6921 2022-09-27T15:31:32.6430191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6922 2022-09-27T15:31:32.6458603Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6923 2022-09-27T15:31:33.8044288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:33.8045290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:33.8082130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:33.8082827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:33.8310629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:33.8311310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:33.8424510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:33.8425098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:33.8462639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:33.8463308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:33.8682161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:33.8682805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:33.8822411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:33.8823025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:33.9191105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:33.9191783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:33.9697706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lh9xm3p 2022-09-27T15:31:33.9698429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lh9xm3p/_remote_module_non_scriptable.py 2022-09-27T15:31:33.9730736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd816hk_x 2022-09-27T15:31:33.9731944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd816hk_x/_remote_module_non_scriptable.py 2022-09-27T15:31:33.9953623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4vg9o7q 2022-09-27T15:31:33.9954981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4vg9o7q/_remote_module_non_scriptable.py 2022-09-27T15:31:34.0464380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbe5ofi4j 2022-09-27T15:31:34.0465115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbe5ofi4j/_remote_module_non_scriptable.py 2022-09-27T15:31:34.2295730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:31:34.2464456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:31:34.2579440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:31:34.3042089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:31:34.9516171Z ok (2.558s) 2022-09-27T15:31:34.9516373Z 2022-09-27T15:31:34.9516856Z ---------------------------------------------------------------------- 2022-09-27T15:31:34.9517315Z Ran 1 test in 2.558s 2022-09-27T15:31:34.9517508Z 2022-09-27T15:31:34.9517606Z OK 2022-09-27T15:31:34.9517758Z 2022-09-27T15:31:34.9517886Z Generating XML reports... 2022-09-27T15:31:34.9552250Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153132.xml 2022-09-27T15:31:36.3366869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:36.3367317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:36.3375196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:36.3375738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:36.4946174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4g40sww1 2022-09-27T15:31:36.4947152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4g40sww1/_remote_module_non_scriptable.py 2022-09-27T15:31:36.7546270Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:36.7556515Z 2022-09-27T15:31:36.7556764Z Running tests... 2022-09-27T15:31:36.7557381Z ---------------------------------------------------------------------- 2022-09-27T15:31:36.9941872Z test_backward_different_tensor_dims (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7163 2022-09-27T15:31:36.9969261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7164 2022-09-27T15:31:36.9996017Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7165 2022-09-27T15:31:37.0023253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7166 2022-09-27T15:31:38.1446991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:38.1447688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:38.1661926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:38.1662487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:38.1836246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:38.1836784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:38.2038047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:38.2038743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:38.2276171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:38.2276793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:38.2476584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:38.2477268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:38.2668473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:38.2669113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:38.2868391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:38.2869081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:38.3119488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo10yfc76 2022-09-27T15:31:38.3120527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo10yfc76/_remote_module_non_scriptable.py 2022-09-27T15:31:38.3322790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5j28h7k 2022-09-27T15:31:38.3323570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5j28h7k/_remote_module_non_scriptable.py 2022-09-27T15:31:38.3949609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbz6krg86 2022-09-27T15:31:38.3950627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbz6krg86/_remote_module_non_scriptable.py 2022-09-27T15:31:38.4158915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yniedfa 2022-09-27T15:31:38.4159982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yniedfa/_remote_module_non_scriptable.py 2022-09-27T15:31:38.5746632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:31:38.5951861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:31:38.6556582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:31:38.6783978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:31:39.4082672Z ok (2.652s) 2022-09-27T15:31:39.4083057Z 2022-09-27T15:31:39.4083578Z ---------------------------------------------------------------------- 2022-09-27T15:31:39.4083896Z Ran 1 test in 2.653s 2022-09-27T15:31:39.4084007Z 2022-09-27T15:31:39.4084067Z OK 2022-09-27T15:31:39.4084160Z 2022-09-27T15:31:39.4084254Z Generating XML reports... 2022-09-27T15:31:39.4119102Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153136.xml 2022-09-27T15:31:40.7711490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:40.7711910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:40.7720356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:40.7720756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:40.9327018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9yylic95 2022-09-27T15:31:40.9329164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9yylic95/_remote_module_non_scriptable.py 2022-09-27T15:31:41.1962425Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:41.1972238Z 2022-09-27T15:31:41.1972362Z Running tests... 2022-09-27T15:31:41.1973055Z ---------------------------------------------------------------------- 2022-09-27T15:31:41.4444816Z test_backward_invalid_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7406 2022-09-27T15:31:41.4471644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7407 2022-09-27T15:31:41.4498863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7408 2022-09-27T15:31:41.4527041Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7409 2022-09-27T15:31:42.5635580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:42.5636206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:42.5938914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:42.5939563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:42.5996722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:42.6299368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:42.6300365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:42.6300940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:42.7251485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4ukrr_b 2022-09-27T15:31:42.7252518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4ukrr_b/_remote_module_non_scriptable.py 2022-09-27T15:31:42.7453782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:42.7570946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:42.7571926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:42.7572507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:42.7574836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa63144q_ 2022-09-27T15:31:42.7577060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa63144q_/_remote_module_non_scriptable.py 2022-09-27T15:31:42.7824764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:42.7825412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:42.7949860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:42.7950477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:42.9086081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3muwrgr0 2022-09-27T15:31:42.9087460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3muwrgr0/_remote_module_non_scriptable.py 2022-09-27T15:31:42.9208859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogbfsmu8 2022-09-27T15:31:42.9209619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogbfsmu8/_remote_module_non_scriptable.py 2022-09-27T15:31:42.9841498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:31:43.0190694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:31:43.1641530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:31:43.1785668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:31:43.8586012Z ok (2.661s) 2022-09-27T15:31:43.8586242Z 2022-09-27T15:31:43.8586713Z ---------------------------------------------------------------------- 2022-09-27T15:31:43.8587090Z Ran 1 test in 2.661s 2022-09-27T15:31:43.8587261Z 2022-09-27T15:31:43.8587354Z OK 2022-09-27T15:31:43.8587499Z 2022-09-27T15:31:43.8587652Z Generating XML reports... 2022-09-27T15:31:43.8623239Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153141.xml 2022-09-27T15:31:45.2326185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:45.2326847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:45.2335237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:45.2335857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:45.3929466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkhylytg 2022-09-27T15:31:45.3930164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkhylytg/_remote_module_non_scriptable.py 2022-09-27T15:31:45.6570854Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:45.6581122Z 2022-09-27T15:31:45.6581415Z Running tests... 2022-09-27T15:31:45.6582032Z ---------------------------------------------------------------------- 2022-09-27T15:31:45.8889241Z test_backward_multiple_output_tensors (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77516 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.230s) 2022-09-27T15:31:45.8890230Z 2022-09-27T15:31:45.8890599Z ---------------------------------------------------------------------- 2022-09-27T15:31:45.8891045Z Ran 1 test in 0.230s 2022-09-27T15:31:45.8891215Z 2022-09-27T15:31:45.8891333Z OK (skipped=1) 2022-09-27T15:31:45.8891510Z 2022-09-27T15:31:45.8891647Z Generating XML reports... 2022-09-27T15:31:45.8913083Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153145.xml 2022-09-27T15:31:47.1897875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:47.1898254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:47.1906324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:47.1906820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:47.3468262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gmd0__e 2022-09-27T15:31:47.3468918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gmd0__e/_remote_module_non_scriptable.py 2022-09-27T15:31:47.6103086Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:47.6112874Z 2022-09-27T15:31:47.6112986Z Running tests... 2022-09-27T15:31:47.6113442Z ---------------------------------------------------------------------- 2022-09-27T15:31:47.8568789Z test_backward_multiple_roots (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7643 2022-09-27T15:31:47.8595446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7644 2022-09-27T15:31:47.8621971Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7645 2022-09-27T15:31:47.8650357Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7646 2022-09-27T15:31:48.9274278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:48.9275114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:48.9472809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:48.9473483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:48.9644207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:48.9644731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:48.9742325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:48.9742970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:48.9844074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:48.9844755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:48.9941480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:48.9942024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:49.0103536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:49.0104165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:49.0297349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:49.0298323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:49.0908520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_vmpnfc 2022-09-27T15:31:49.0909514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_vmpnfc/_remote_module_non_scriptable.py 2022-09-27T15:31:49.1103051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39s0zt2r 2022-09-27T15:31:49.1104355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39s0zt2r/_remote_module_non_scriptable.py 2022-09-27T15:31:49.1371448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5tnxr2q 2022-09-27T15:31:49.1372721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5tnxr2q/_remote_module_non_scriptable.py 2022-09-27T15:31:49.1548193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_bpj0s6 2022-09-27T15:31:49.1549258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_bpj0s6/_remote_module_non_scriptable.py 2022-09-27T15:31:49.3460331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:31:49.3666537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:31:49.3955671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:31:49.4104890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:31:50.1707189Z ok (2.559s) 2022-09-27T15:31:50.1707355Z 2022-09-27T15:31:50.1707710Z ---------------------------------------------------------------------- 2022-09-27T15:31:50.1708029Z Ran 1 test in 2.559s 2022-09-27T15:31:50.1708153Z 2022-09-27T15:31:50.1708214Z OK 2022-09-27T15:31:50.1708308Z 2022-09-27T15:31:50.1708395Z Generating XML reports... 2022-09-27T15:31:50.1743631Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153147.xml 2022-09-27T15:31:51.5335504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:51.5335909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:51.5343958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:51.5344312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:51.6935110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybz1v3k5 2022-09-27T15:31:51.6936416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybz1v3k5/_remote_module_non_scriptable.py 2022-09-27T15:31:51.9566081Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:51.9575718Z 2022-09-27T15:31:51.9575834Z Running tests... 2022-09-27T15:31:51.9576275Z ---------------------------------------------------------------------- 2022-09-27T15:31:52.1916695Z test_backward_multiple_round_trips (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77453 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.234s) 2022-09-27T15:31:52.1918601Z 2022-09-27T15:31:52.1918925Z ---------------------------------------------------------------------- 2022-09-27T15:31:52.1919260Z Ran 1 test in 0.234s 2022-09-27T15:31:52.1919423Z 2022-09-27T15:31:52.1919534Z OK (skipped=1) 2022-09-27T15:31:52.1919712Z 2022-09-27T15:31:52.1919839Z Generating XML reports... 2022-09-27T15:31:52.1940992Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153151.xml 2022-09-27T15:31:53.5165828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:53.5166499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:53.5174222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:53.5174826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:53.6770751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_x_66aar 2022-09-27T15:31:53.6771589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_x_66aar/_remote_module_non_scriptable.py 2022-09-27T15:31:53.9357305Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:53.9366948Z 2022-09-27T15:31:53.9367205Z Running tests... 2022-09-27T15:31:53.9367542Z ---------------------------------------------------------------------- 2022-09-27T15:31:54.1823078Z test_backward_no_grad_on_tensor (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7896 2022-09-27T15:31:54.1850935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7897 2022-09-27T15:31:54.1877844Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7898 2022-09-27T15:31:54.1905958Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7899 2022-09-27T15:31:55.2802840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:55.2803477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:55.3063074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:55.3063810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:55.3138849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:55.3139363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:55.3180808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:55.3247683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:55.3248603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:55.3249197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:55.3451531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:55.3452196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:55.3528423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:55.3529089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:55.3623229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:55.3624020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:55.4459546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcs2e0ln 2022-09-27T15:31:55.4460564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcs2e0ln/_remote_module_non_scriptable.py 2022-09-27T15:31:55.4720536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20fz19np 2022-09-27T15:31:55.4721663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20fz19np/_remote_module_non_scriptable.py 2022-09-27T15:31:55.4803051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3v3sslh 2022-09-27T15:31:55.4804062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3v3sslh/_remote_module_non_scriptable.py 2022-09-27T15:31:55.4892309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsad3506c 2022-09-27T15:31:55.4893787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsad3506c/_remote_module_non_scriptable.py 2022-09-27T15:31:55.7211331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:31:55.7275175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:31:55.7424524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:31:55.7461033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:31:56.3962613Z ok (2.459s) 2022-09-27T15:31:56.3962886Z 2022-09-27T15:31:56.3963384Z ---------------------------------------------------------------------- 2022-09-27T15:31:56.3963642Z Ran 1 test in 2.459s 2022-09-27T15:31:56.3963760Z 2022-09-27T15:31:56.3963818Z OK 2022-09-27T15:31:56.3963911Z 2022-09-27T15:31:56.3964011Z Generating XML reports... 2022-09-27T15:31:56.3999782Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153153.xml 2022-09-27T15:31:57.7780514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:57.7780921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:57.7789816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:57.7790255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:57.9377081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp533psq_1 2022-09-27T15:31:57.9378532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp533psq_1/_remote_module_non_scriptable.py 2022-09-27T15:31:58.2000314Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:31:58.2010602Z 2022-09-27T15:31:58.2010736Z Running tests... 2022-09-27T15:31:58.2011125Z ---------------------------------------------------------------------- 2022-09-27T15:31:58.4447489Z test_backward_node_failure (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8139 2022-09-27T15:31:58.4472492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8140 2022-09-27T15:31:58.4499105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8141 2022-09-27T15:31:58.4527472Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8142 2022-09-27T15:31:59.5695298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:59.5695974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:59.5763026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:59.5763668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:59.6059273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:59.6059911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:59.6131540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:59.6170548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:59.6171459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:59.6172375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:59.6436542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:31:59.6437465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:31:59.6550800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:59.6551366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:59.6793968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:31:59.6794805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:31:59.7319899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoj6cs6ls 2022-09-27T15:31:59.7320842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoj6cs6ls/_remote_module_non_scriptable.py 2022-09-27T15:31:59.7402154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc97s342z 2022-09-27T15:31:59.7403556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc97s342z/_remote_module_non_scriptable.py 2022-09-27T15:31:59.7812751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxthx_y8q 2022-09-27T15:31:59.7813529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxthx_y8q/_remote_module_non_scriptable.py 2022-09-27T15:31:59.8054974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdi_7mbuk 2022-09-27T15:31:59.8055732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdi_7mbuk/_remote_module_non_scriptable.py 2022-09-27T15:31:59.9869621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:31:59.9971704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:32:00.0555664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:32:00.0636679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:32:00.3008489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:32:00.3211538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:32:00.3315475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:32:00.3316659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:32:00.3318474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:32:00.3322291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:32:00.3324122Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:32:00.3325280Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:32:00.3493639Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:32:00.3494769Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:32:00.3496335Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.3497559Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.3498498Z [E container.cpp:256] Could not release Dist Autograd Context on node 0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:32:00.3499350Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:32:00.3500217Z [E container.cpp:256] Could not release Dist Autograd Context on node 2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:32:00.3501024Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.3501898Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.3504448Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #2 to worker1: ECONNRESET: connection reset by peer (this error originated at tensorpipe/transport/uv/connection_impl.cc:132) 2022-09-27T15:32:00.3505764Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #3 to worker1: ECONNRESET: connection reset by peer (this error originated at tensorpipe/transport/uv/connection_impl.cc:132) 2022-09-27T15:32:00.3510036Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #5 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.3510978Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #4 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.3522406Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #6 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.3524054Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #7 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.3526240Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #5 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.3546215Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #8 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.3547073Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #9 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.3548629Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.3552324Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:32:00.3572785Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #6 to worker1: ECONNRESET: connection reset by peer (this error originated at tensorpipe/transport/uv/connection_impl.cc:132) 2022-09-27T15:32:00.3574085Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #7 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.3574888Z [E container.cpp:256] Could not release Dist Autograd Context on node 3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:00.7585009Z ok (2.557s) 2022-09-27T15:32:00.7585256Z 2022-09-27T15:32:00.7585636Z ---------------------------------------------------------------------- 2022-09-27T15:32:00.7585903Z Ran 1 test in 2.557s 2022-09-27T15:32:00.7586021Z 2022-09-27T15:32:00.7586083Z OK 2022-09-27T15:32:00.7586180Z 2022-09-27T15:32:00.7586262Z Generating XML reports... 2022-09-27T15:32:00.7621120Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153158.xml 2022-09-27T15:32:02.1576229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:02.1576860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:02.1585504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:02.1585894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:02.3150270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvtu_5c_ 2022-09-27T15:32:02.3151124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvtu_5c_/_remote_module_non_scriptable.py 2022-09-27T15:32:02.5786223Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:32:02.5795254Z 2022-09-27T15:32:02.5795376Z Running tests... 2022-09-27T15:32:02.5795996Z ---------------------------------------------------------------------- 2022-09-27T15:32:02.8242007Z test_backward_node_failure_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8384 2022-09-27T15:32:02.8268737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8385 2022-09-27T15:32:02.8296769Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8386 2022-09-27T15:32:02.8324491Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8387 2022-09-27T15:32:03.9974675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:03.9975311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:04.0171676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:04.0172366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:04.0353223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:04.0353807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:04.0386308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:04.0386818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:04.0387450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:04.0387900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:04.0557557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:04.0558240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:04.0764459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:04.0765125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:04.0766503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:04.0767147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:04.1613078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxucy_fa 2022-09-27T15:32:04.1613836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxucy_fa/_remote_module_non_scriptable.py 2022-09-27T15:32:04.1824658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp192_gk27 2022-09-27T15:32:04.1825463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp192_gk27/_remote_module_non_scriptable.py 2022-09-27T15:32:04.2024933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahwjy50p 2022-09-27T15:32:04.2025685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpde8utyak 2022-09-27T15:32:04.2027613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahwjy50p/_remote_module_non_scriptable.py 2022-09-27T15:32:04.2028353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpde8utyak/_remote_module_non_scriptable.py 2022-09-27T15:32:04.4164444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:32:04.4387868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:32:04.4561099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:32:04.4576089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:32:04.7246094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:32:04.7346176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:32:04.7448769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:32:04.7449517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:32:04.7450794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:32:04.7452835Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:32:04.7457729Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:32:04.7459706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:32:04.7613522Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:04.7615277Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:04.7616710Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:32:04.7617528Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:32:04.7618360Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:04.7619446Z [W tensorpipe_agent.cpp:916] RPC agent for worker3 encountered error when sending outgoing request #4 to worker2: ECONNRESET: connection reset by peer (this error originated at tensorpipe/transport/uv/connection_impl.cc:132) 2022-09-27T15:32:04.7621746Z [E container.cpp:256] Could not release Dist Autograd Context on node 3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:32:04.7639143Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #5 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:04.7667945Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #10 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:04.7669926Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #7 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:04.7671204Z [W tensorpipe_agent.cpp:625] RPC agent for worker0 won't send response to request #6 to worker1, as the agent is shutting down 2022-09-27T15:32:04.7679582Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:32:04.7680656Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:32:04.7681701Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:04.7682774Z [W tensorpipe_agent.cpp:940] RPC agent for worker1 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:04.7683798Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:04.7709433Z [W tensorpipe_agent.cpp:916] RPC agent for worker3 encountered error when sending outgoing request #6 to worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:04.7711329Z [E container.cpp:256] Could not release Dist Autograd Context on node 0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:04.7712282Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #8 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:32:05.1381399Z ok (2.558s) 2022-09-27T15:32:05.1381594Z 2022-09-27T15:32:05.1381898Z ---------------------------------------------------------------------- 2022-09-27T15:32:05.1382164Z Ran 1 test in 2.559s 2022-09-27T15:32:05.1382283Z 2022-09-27T15:32:05.1382342Z OK 2022-09-27T15:32:05.1382422Z 2022-09-27T15:32:05.1382530Z Generating XML reports... 2022-09-27T15:32:05.1417865Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153202.xml 2022-09-27T15:32:06.5331890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:06.5332271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:06.5341053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:06.5341444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:06.6925976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0w5ksv0y 2022-09-27T15:32:06.6928049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0w5ksv0y/_remote_module_non_scriptable.py 2022-09-27T15:32:06.9551586Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:32:06.9561698Z 2022-09-27T15:32:06.9561819Z Running tests... 2022-09-27T15:32:06.9562413Z ---------------------------------------------------------------------- 2022-09-27T15:32:07.1985624Z test_backward_python_udf_error (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8629 2022-09-27T15:32:07.2012588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8630 2022-09-27T15:32:07.2040610Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8631 2022-09-27T15:32:07.2074844Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8632 2022-09-27T15:32:08.3156652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:08.3157275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:08.3330780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:08.3331412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:08.3428924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:08.3429619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:08.3541321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:08.3541854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:08.3599977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:08.3600636Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:08.3710779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:08.3711508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:08.3815354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:08.3816043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:08.3958023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:08.3958677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:08.4812802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp0pjqyjv 2022-09-27T15:32:08.4813642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp0pjqyjv/_remote_module_non_scriptable.py 2022-09-27T15:32:08.4986588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi01xsca_ 2022-09-27T15:32:08.4987410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi01xsca_/_remote_module_non_scriptable.py 2022-09-27T15:32:08.5083924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpndiqvjyq 2022-09-27T15:32:08.5085332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpndiqvjyq/_remote_module_non_scriptable.py 2022-09-27T15:32:08.5226540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqs7_4uql 2022-09-27T15:32:08.5227460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqs7_4uql/_remote_module_non_scriptable.py 2022-09-27T15:32:08.7380534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:32:08.7559550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:32:08.7634107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:32:08.7801935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:32:09.4131227Z ok (2.457s) 2022-09-27T15:32:09.4131823Z 2022-09-27T15:32:09.4132327Z ---------------------------------------------------------------------- 2022-09-27T15:32:09.4132597Z Ran 1 test in 2.457s 2022-09-27T15:32:09.4132715Z 2022-09-27T15:32:09.4132774Z OK 2022-09-27T15:32:09.4132868Z 2022-09-27T15:32:09.4132972Z Generating XML reports... 2022-09-27T15:32:09.4167924Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153206.xml 2022-09-27T15:32:10.7840194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:10.7840565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:10.7848568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:10.7849089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:10.9406461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmegbc6op 2022-09-27T15:32:10.9407723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmegbc6op/_remote_module_non_scriptable.py 2022-09-27T15:32:11.2016233Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:32:11.2025643Z 2022-09-27T15:32:11.2025787Z Running tests... 2022-09-27T15:32:11.2026267Z ---------------------------------------------------------------------- 2022-09-27T15:32:11.4491276Z test_backward_rref (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8872 2022-09-27T15:32:11.4518275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8873 2022-09-27T15:32:11.4545327Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8874 2022-09-27T15:32:11.4574237Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8875 2022-09-27T15:32:12.5318540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:12.5319287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:12.5684013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:12.5684587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:12.5775111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:12.5775799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:12.5827386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:12.5828051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:12.5944315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:12.5944861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:12.6143141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:12.6143870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:12.6198543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:12.6199205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:12.6317179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:12.6317798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:12.6945508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuw0tdb_j 2022-09-27T15:32:12.6946421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuw0tdb_j/_remote_module_non_scriptable.py 2022-09-27T15:32:12.7401589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tjve74h 2022-09-27T15:32:12.7402676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tjve74h/_remote_module_non_scriptable.py 2022-09-27T15:32:12.7468005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbozg32a 2022-09-27T15:32:12.7468999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbozg32a/_remote_module_non_scriptable.py 2022-09-27T15:32:12.7592611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5g58ik4k 2022-09-27T15:32:12.7593900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5g58ik4k/_remote_module_non_scriptable.py 2022-09-27T15:32:12.9485506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:32:12.9954823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:32:13.0036981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:32:13.0154934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:32:13.6630474Z ok (2.460s) 2022-09-27T15:32:13.6630708Z 2022-09-27T15:32:13.6631223Z ---------------------------------------------------------------------- 2022-09-27T15:32:13.6631596Z Ran 1 test in 2.460s 2022-09-27T15:32:13.6631716Z 2022-09-27T15:32:13.6631776Z OK 2022-09-27T15:32:13.6631868Z 2022-09-27T15:32:13.6631963Z Generating XML reports... 2022-09-27T15:32:13.6666838Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153211.xml 2022-09-27T15:32:15.0269596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:15.0269975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:15.0278663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:15.0279075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:15.1871000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocuycmct 2022-09-27T15:32:15.1871977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocuycmct/_remote_module_non_scriptable.py 2022-09-27T15:32:15.4488336Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:32:15.4497461Z 2022-09-27T15:32:15.4497710Z Running tests... 2022-09-27T15:32:15.4498330Z ---------------------------------------------------------------------- 2022-09-27T15:32:15.6938028Z test_backward_rref_multi (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9115 2022-09-27T15:32:15.6969481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9116 2022-09-27T15:32:15.6997224Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9117 2022-09-27T15:32:15.7026318Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9118 2022-09-27T15:32:16.7912639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:16.7913327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:16.7914136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:16.7914967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:16.8247479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:16.8248196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:16.8274724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:16.8275435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:16.8280545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:16.8281096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:16.8608910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:16.8609555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:16.8690190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:16.8690817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:16.9055382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:16.9056046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:16.9545440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbrsuobu 2022-09-27T15:32:16.9546482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbrsuobu/_remote_module_non_scriptable.py 2022-09-27T15:32:16.9547236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvzkeg303 2022-09-27T15:32:16.9549651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvzkeg303/_remote_module_non_scriptable.py 2022-09-27T15:32:16.9869022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8wpt0x1n 2022-09-27T15:32:16.9871541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8wpt0x1n/_remote_module_non_scriptable.py 2022-09-27T15:32:17.0326350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkuff4fxg 2022-09-27T15:32:17.0327059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkuff4fxg/_remote_module_non_scriptable.py 2022-09-27T15:32:17.2125353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:32:17.2150239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:32:17.2484434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:32:17.2939919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:32:17.9084584Z ok (2.458s) 2022-09-27T15:32:17.9084827Z 2022-09-27T15:32:17.9085260Z ---------------------------------------------------------------------- 2022-09-27T15:32:17.9085666Z Ran 1 test in 2.459s 2022-09-27T15:32:17.9085834Z 2022-09-27T15:32:17.9085932Z OK 2022-09-27T15:32:17.9086103Z 2022-09-27T15:32:17.9086249Z Generating XML reports... 2022-09-27T15:32:17.9121834Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153215.xml 2022-09-27T15:32:19.2725705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:19.2726072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:19.2733847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:19.2734213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:19.4366515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphw94u483 2022-09-27T15:32:19.4367786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphw94u483/_remote_module_non_scriptable.py 2022-09-27T15:32:19.7026067Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:32:19.7036448Z 2022-09-27T15:32:19.7036541Z Running tests... 2022-09-27T15:32:19.7037118Z ---------------------------------------------------------------------- 2022-09-27T15:32:19.9427611Z test_backward_rref_nested (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9358 2022-09-27T15:32:19.9453288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9359 2022-09-27T15:32:19.9480292Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9360 2022-09-27T15:32:19.9507466Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9361 2022-09-27T15:32:20.9945995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:20.9946627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:21.0133292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:21.0133954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:21.0182871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:21.0183534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:21.0307969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:21.0308583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:21.0505767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:21.0506420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:21.0535021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:21.0535687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:21.0556417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:21.0557497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:21.0889248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:21.0889932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:21.1576545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiyqu2e6z 2022-09-27T15:32:21.1577581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiyqu2e6z/_remote_module_non_scriptable.py 2022-09-27T15:32:21.1773054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1gyjhhd 2022-09-27T15:32:21.1773967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1gyjhhd/_remote_module_non_scriptable.py 2022-09-27T15:32:21.1812131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnps1fam3 2022-09-27T15:32:21.1814080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnps1fam3/_remote_module_non_scriptable.py 2022-09-27T15:32:21.2147682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7xnt01l 2022-09-27T15:32:21.2148718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7xnt01l/_remote_module_non_scriptable.py 2022-09-27T15:32:21.4111723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:32:21.4303493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:32:21.4328112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:32:21.4700825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:32:22.1564107Z ok (2.452s) 2022-09-27T15:32:22.1564291Z 2022-09-27T15:32:22.1564670Z ---------------------------------------------------------------------- 2022-09-27T15:32:22.1564922Z Ran 1 test in 2.453s 2022-09-27T15:32:22.1565040Z 2022-09-27T15:32:22.1565100Z OK 2022-09-27T15:32:22.1565179Z 2022-09-27T15:32:22.1565275Z Generating XML reports... 2022-09-27T15:32:22.1601800Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153219.xml 2022-09-27T15:32:23.5363712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:23.5364090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:23.5371985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:23.5372347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:23.6924760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn75hlm8c 2022-09-27T15:32:23.6925615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn75hlm8c/_remote_module_non_scriptable.py 2022-09-27T15:32:23.9537747Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:32:23.9547731Z 2022-09-27T15:32:23.9548143Z Running tests... 2022-09-27T15:32:23.9548757Z ---------------------------------------------------------------------- 2022-09-27T15:32:24.1968123Z test_backward_simple (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9601 2022-09-27T15:32:24.1994961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9602 2022-09-27T15:32:24.2022003Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9603 2022-09-27T15:32:24.2050235Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9604 2022-09-27T15:32:25.2964067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:25.2964748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:25.3206429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:25.3207069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:25.3210509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:25.3211139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:25.3350854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:25.3351398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:25.3468748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:25.3469353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:25.3580936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:25.3581533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:25.3587275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:25.3587917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:25.3823531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:25.3824234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:25.4609540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmqzot8mb 2022-09-27T15:32:25.4610840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmqzot8mb/_remote_module_non_scriptable.py 2022-09-27T15:32:25.4841143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzo1egdkw 2022-09-27T15:32:25.4841895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzo1egdkw/_remote_module_non_scriptable.py 2022-09-27T15:32:25.4850533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnou9q5h 2022-09-27T15:32:25.4852225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnou9q5h/_remote_module_non_scriptable.py 2022-09-27T15:32:25.5090480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt10yzj8c 2022-09-27T15:32:25.5091209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt10yzj8c/_remote_module_non_scriptable.py 2022-09-27T15:32:25.7140697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:32:25.7372164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:32:25.7405793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:32:25.7666526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:32:26.4106296Z ok (2.455s) 2022-09-27T15:32:26.4106571Z 2022-09-27T15:32:26.4107088Z ---------------------------------------------------------------------- 2022-09-27T15:32:26.4107385Z Ran 1 test in 2.456s 2022-09-27T15:32:26.4107505Z 2022-09-27T15:32:26.4107563Z OK 2022-09-27T15:32:26.4107641Z 2022-09-27T15:32:26.4107734Z Generating XML reports... 2022-09-27T15:32:26.4141907Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153223.xml 2022-09-27T15:32:27.7641052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:27.7641419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:27.7649551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:27.7649913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:27.9231949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyplp3nve 2022-09-27T15:32:27.9233305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyplp3nve/_remote_module_non_scriptable.py 2022-09-27T15:32:28.1821329Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:32:28.1830826Z 2022-09-27T15:32:28.1830934Z Running tests... 2022-09-27T15:32:28.1831386Z ---------------------------------------------------------------------- 2022-09-27T15:32:28.4283751Z test_backward_simple_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9844 2022-09-27T15:32:28.4309979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9845 2022-09-27T15:32:28.4337462Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9846 2022-09-27T15:32:28.4365425Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9847 2022-09-27T15:32:29.5478344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:29.5478971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:29.5554404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:29.5555203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:29.5788641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:29.5789289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:29.5795909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:29.5796529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:29.5855856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:29.5856395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:29.5935828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:29.5936371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:29.6174449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:29.6175086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:29.6183349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:29.6183890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:29.7122477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpav0b_1yp 2022-09-27T15:32:29.7123330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpav0b_1yp/_remote_module_non_scriptable.py 2022-09-27T15:32:29.7191620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0_xd57_ 2022-09-27T15:32:29.7192970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0_xd57_/_remote_module_non_scriptable.py 2022-09-27T15:32:29.7437440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1jtxun1 2022-09-27T15:32:29.7438145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1jtxun1/_remote_module_non_scriptable.py 2022-09-27T15:32:29.7448075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcvyo78q 2022-09-27T15:32:29.7449697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcvyo78q/_remote_module_non_scriptable.py 2022-09-27T15:32:29.9675977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:32:29.9737303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:32:29.9997202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:32:30.0216148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:32:30.6420892Z ok (2.459s) 2022-09-27T15:32:30.6421167Z 2022-09-27T15:32:30.6421677Z ---------------------------------------------------------------------- 2022-09-27T15:32:30.6422000Z Ran 1 test in 2.459s 2022-09-27T15:32:30.6422119Z 2022-09-27T15:32:30.6422177Z OK 2022-09-27T15:32:30.6422268Z 2022-09-27T15:32:30.6422364Z Generating XML reports... 2022-09-27T15:32:30.6460403Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153228.xml 2022-09-27T15:32:32.0004264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:32.0004645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:32.0012865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:32.0013226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:32.1556664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5zxh03h_ 2022-09-27T15:32:32.1557480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5zxh03h_/_remote_module_non_scriptable.py 2022-09-27T15:32:32.4184123Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:32:32.4194341Z 2022-09-27T15:32:32.4194869Z Running tests... 2022-09-27T15:32:32.4195440Z ---------------------------------------------------------------------- 2022-09-27T15:32:32.6610942Z test_backward_simple_script_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10087 2022-09-27T15:32:32.6637215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10088 2022-09-27T15:32:32.6663975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10089 2022-09-27T15:32:32.6692207Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10090 2022-09-27T15:32:33.8638834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:33.8639525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:33.8765834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:33.8766531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:33.8851816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:33.8852401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:33.8853995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:33.8854587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:33.9018060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:33.9018589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:33.9156370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:33.9157042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:33.9241108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:33.9241815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:33.9246314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:33.9246973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:34.0382301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyna47pnl 2022-09-27T15:32:34.0383144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyna47pnl/_remote_module_non_scriptable.py 2022-09-27T15:32:34.0409004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkq87gtnq 2022-09-27T15:32:34.0411173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkq87gtnq/_remote_module_non_scriptable.py 2022-09-27T15:32:34.0496847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptd2ipndk 2022-09-27T15:32:34.0497999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptd2ipndk/_remote_module_non_scriptable.py 2022-09-27T15:32:34.0512935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7xe031x 2022-09-27T15:32:34.0514938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7xe031x/_remote_module_non_scriptable.py 2022-09-27T15:32:34.2962402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:32:34.3064592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:32:34.3073304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:32:34.3108064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:32:35.1849341Z ok (2.765s) 2022-09-27T15:32:35.1849566Z 2022-09-27T15:32:35.1849895Z ---------------------------------------------------------------------- 2022-09-27T15:32:35.1850145Z Ran 1 test in 2.765s 2022-09-27T15:32:35.1850249Z 2022-09-27T15:32:35.1850310Z OK 2022-09-27T15:32:35.1850400Z 2022-09-27T15:32:35.1850493Z Generating XML reports... 2022-09-27T15:32:35.1885859Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153232.xml 2022-09-27T15:32:36.5603693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:36.5604059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:36.5612435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:36.5612801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:36.7185009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1zi0ghh 2022-09-27T15:32:36.7186461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1zi0ghh/_remote_module_non_scriptable.py 2022-09-27T15:32:36.9766947Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:32:36.9776795Z 2022-09-27T15:32:36.9776916Z Running tests... 2022-09-27T15:32:36.9777408Z ---------------------------------------------------------------------- 2022-09-27T15:32:37.2177562Z test_backward_simple_self (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10330 2022-09-27T15:32:37.2204104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10331 2022-09-27T15:32:37.2231748Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10332 2022-09-27T15:32:37.2267273Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10333 2022-09-27T15:32:38.3127234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:38.3127879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:38.3213776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:38.3214387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:38.3487515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:38.3488215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:38.3520903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:38.3604351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:38.3605264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:38.3605901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:38.3862985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:38.3863653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:38.3888831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:38.3889513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:38.4248784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:38.4249776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:38.4796302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4cn_5rf7 2022-09-27T15:32:38.4797589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4cn_5rf7/_remote_module_non_scriptable.py 2022-09-27T15:32:38.4880044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21mhg3sn 2022-09-27T15:32:38.4881097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21mhg3sn/_remote_module_non_scriptable.py 2022-09-27T15:32:38.5133327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2p1aesh 2022-09-27T15:32:38.5134040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2p1aesh/_remote_module_non_scriptable.py 2022-09-27T15:32:38.5523649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vl3y4ib 2022-09-27T15:32:38.5524493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vl3y4ib/_remote_module_non_scriptable.py 2022-09-27T15:32:38.7397215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:32:38.7477685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:32:38.7748037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:32:38.8159466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:32:39.5325487Z ok (2.554s) 2022-09-27T15:32:39.5325720Z 2022-09-27T15:32:39.5326099Z ---------------------------------------------------------------------- 2022-09-27T15:32:39.5326405Z Ran 1 test in 2.555s 2022-09-27T15:32:39.5326527Z 2022-09-27T15:32:39.5326576Z OK 2022-09-27T15:32:39.5326673Z 2022-09-27T15:32:39.5326813Z Generating XML reports... 2022-09-27T15:32:39.5362513Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153236.xml 2022-09-27T15:32:40.9106410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:40.9106980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:40.9115829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:40.9116289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:41.0701140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifsocxw5 2022-09-27T15:32:41.0702272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifsocxw5/_remote_module_non_scriptable.py 2022-09-27T15:32:41.3323820Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:32:41.3333334Z 2022-09-27T15:32:41.3333423Z Running tests... 2022-09-27T15:32:41.3334320Z ---------------------------------------------------------------------- 2022-09-27T15:32:41.5727844Z test_backward_unused_send_function (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10573 2022-09-27T15:32:41.5755016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10574 2022-09-27T15:32:41.5782672Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10575 2022-09-27T15:32:41.5811813Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10576 2022-09-27T15:32:42.6643327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:42.6644265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:42.6821804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:42.6822804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:42.6913636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:42.6914199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:42.7014894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:42.7015414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:42.7130788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:42.7131431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:42.7215847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:42.7216505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:42.7308980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:42.7309629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:42.7501618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:42.7502261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:42.8284594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmnw46m4 2022-09-27T15:32:42.8285437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmnw46m4/_remote_module_non_scriptable.py 2022-09-27T15:32:42.8482900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbajxq4e 2022-09-27T15:32:42.8483669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbajxq4e/_remote_module_non_scriptable.py 2022-09-27T15:32:42.8574553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpir2uuc8s 2022-09-27T15:32:42.8575471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpir2uuc8s/_remote_module_non_scriptable.py 2022-09-27T15:32:42.8765029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1t4njnm8 2022-09-27T15:32:42.8765707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1t4njnm8/_remote_module_non_scriptable.py 2022-09-27T15:32:43.0834682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:32:43.1020692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:32:43.1114830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:32:43.1313520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:32:53.7018383Z ok (12.368s) 2022-09-27T15:32:53.7018670Z 2022-09-27T15:32:53.7019002Z ---------------------------------------------------------------------- 2022-09-27T15:32:53.7019298Z Ran 1 test in 12.368s 2022-09-27T15:32:53.7019414Z 2022-09-27T15:32:53.7019473Z OK 2022-09-27T15:32:53.7019563Z 2022-09-27T15:32:53.7019644Z Generating XML reports... 2022-09-27T15:32:53.7055101Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153241.xml 2022-09-27T15:32:55.0569071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:55.0569496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:55.0577549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:55.0577918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:55.2111481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5gu9uns 2022-09-27T15:32:55.2112603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5gu9uns/_remote_module_non_scriptable.py 2022-09-27T15:32:55.4726781Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:32:55.4735871Z 2022-09-27T15:32:55.4735969Z Running tests... 2022-09-27T15:32:55.4736380Z ---------------------------------------------------------------------- 2022-09-27T15:32:55.7077173Z test_backward_unused_tensors (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77556 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.234s) 2022-09-27T15:32:55.7077675Z 2022-09-27T15:32:55.7077962Z ---------------------------------------------------------------------- 2022-09-27T15:32:55.7078196Z Ran 1 test in 0.234s 2022-09-27T15:32:55.7078307Z 2022-09-27T15:32:55.7078378Z OK (skipped=1) 2022-09-27T15:32:55.7078485Z 2022-09-27T15:32:55.7078601Z Generating XML reports... 2022-09-27T15:32:55.7100275Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153255.xml 2022-09-27T15:32:57.0222050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:57.0222429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:57.0231471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:57.0231846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:57.1832003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftilx7hi 2022-09-27T15:32:57.1832827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftilx7hi/_remote_module_non_scriptable.py 2022-09-27T15:32:57.4435203Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:32:57.4444531Z 2022-09-27T15:32:57.4444663Z Running tests... 2022-09-27T15:32:57.4445231Z ---------------------------------------------------------------------- 2022-09-27T15:32:57.6865404Z test_backward_verify_hooks (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10830 2022-09-27T15:32:57.6893570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10831 2022-09-27T15:32:57.6921775Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10832 2022-09-27T15:32:57.6950414Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10833 2022-09-27T15:32:58.8023686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:58.8024372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:58.8025178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:58.8025787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:58.8232236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:58.8232759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:58.8233361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:32:58.8234119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:32:58.8421818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:58.8422754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:58.8423420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:58.8423906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:58.8616594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:58.8617169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:58.8618837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:32:58.8619452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:32:58.9693257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fnwgyxg 2022-09-27T15:32:58.9694082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fnwgyxg/_remote_module_non_scriptable.py 2022-09-27T15:32:58.9695675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplamghaab 2022-09-27T15:32:58.9698266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplamghaab/_remote_module_non_scriptable.py 2022-09-27T15:32:58.9876266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd54ntvar 2022-09-27T15:32:58.9876955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd54ntvar/_remote_module_non_scriptable.py 2022-09-27T15:32:58.9880941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgo5te_ou 2022-09-27T15:32:58.9883145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgo5te_ou/_remote_module_non_scriptable.py 2022-09-27T15:32:59.2317209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:32:59.2464540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:32:59.2475209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:32:59.2497321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:00.0008767Z ok (2.556s) 2022-09-27T15:33:00.0008998Z 2022-09-27T15:33:00.0009435Z ---------------------------------------------------------------------- 2022-09-27T15:33:00.0009834Z Ran 1 test in 2.556s 2022-09-27T15:33:00.0010014Z 2022-09-27T15:33:00.0010105Z OK 2022-09-27T15:33:00.0010263Z 2022-09-27T15:33:00.0010398Z Generating XML reports... 2022-09-27T15:33:00.0046062Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153257.xml 2022-09-27T15:33:01.4144718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:01.4145113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:01.4153195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:01.4153554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:01.5740090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9ct5waa 2022-09-27T15:33:01.5741221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9ct5waa/_remote_module_non_scriptable.py 2022-09-27T15:33:01.8407785Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:01.8417558Z 2022-09-27T15:33:01.8417850Z Running tests... 2022-09-27T15:33:01.8418758Z ---------------------------------------------------------------------- 2022-09-27T15:33:02.0865648Z test_backward_without_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11073 2022-09-27T15:33:02.0894141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11074 2022-09-27T15:33:02.0921948Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11075 2022-09-27T15:33:02.0950527Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11076 2022-09-27T15:33:03.2288369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:03.2289034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:03.2330673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:03.2331352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:03.2620190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:03.2620907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:03.2663998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:03.2664639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:03.2705088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:03.2705695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:03.2811490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:03.2812079Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:03.2982535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:03.2983209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:03.3227780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:03.3228238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:03.3957466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3hjts4j8 2022-09-27T15:33:03.3958152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3hjts4j8/_remote_module_non_scriptable.py 2022-09-27T15:33:03.3991829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps7s39x_w 2022-09-27T15:33:03.3993228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps7s39x_w/_remote_module_non_scriptable.py 2022-09-27T15:33:03.4244204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5iqxqzyb 2022-09-27T15:33:03.4244852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5iqxqzyb/_remote_module_non_scriptable.py 2022-09-27T15:33:03.4567922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_e8e5plv 2022-09-27T15:33:03.4568816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_e8e5plv/_remote_module_non_scriptable.py 2022-09-27T15:33:03.6538693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:33:03.6539638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:03.6811975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:33:03.7110595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:33:04.4008680Z ok (2.559s) 2022-09-27T15:33:04.4008973Z 2022-09-27T15:33:04.4009507Z ---------------------------------------------------------------------- 2022-09-27T15:33:04.4009763Z Ran 1 test in 2.559s 2022-09-27T15:33:04.4009946Z 2022-09-27T15:33:04.4009993Z OK 2022-09-27T15:33:04.4010085Z 2022-09-27T15:33:04.4010184Z Generating XML reports... 2022-09-27T15:33:04.4046845Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153301.xml 2022-09-27T15:33:05.8043262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:05.8043648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:05.8051854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:05.8052240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:05.9613813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaz0508u3 2022-09-27T15:33:05.9614624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaz0508u3/_remote_module_non_scriptable.py 2022-09-27T15:33:06.2210438Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:06.2219993Z 2022-09-27T15:33:06.2220386Z Running tests... 2022-09-27T15:33:06.2220788Z ---------------------------------------------------------------------- 2022-09-27T15:33:06.4667910Z test_backward_without_rpc (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11300 2022-09-27T15:33:06.4695603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11301 2022-09-27T15:33:06.4723404Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11302 2022-09-27T15:33:06.4751976Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11303 2022-09-27T15:33:07.6011252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:07.6011887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:07.6232366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:07.6233007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:07.6393496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:07.6394152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:07.6600160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:07.6600823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:07.6609791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:07.6620566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:07.6622037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:07.6622712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:07.6985320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:07.6985953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:07.6999190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:07.6999863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:07.7678488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7n33fcwf 2022-09-27T15:33:07.7679575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7n33fcwf/_remote_module_non_scriptable.py 2022-09-27T15:33:07.7891931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpam3fcxnf 2022-09-27T15:33:07.7892969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpam3fcxnf/_remote_module_non_scriptable.py 2022-09-27T15:33:07.8251584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4xbnzhu 2022-09-27T15:33:07.8252332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4xbnzhu/_remote_module_non_scriptable.py 2022-09-27T15:33:07.8261581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa67l071c 2022-09-27T15:33:07.8263226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa67l071c/_remote_module_non_scriptable.py 2022-09-27T15:33:08.0259131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:33:08.0477473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:33:08.0812307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:08.0819248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:33:08.6808518Z ok (2.458s) 2022-09-27T15:33:08.6808730Z 2022-09-27T15:33:08.6809097Z ---------------------------------------------------------------------- 2022-09-27T15:33:08.6809349Z Ran 1 test in 2.459s 2022-09-27T15:33:08.6809450Z 2022-09-27T15:33:08.6809510Z OK 2022-09-27T15:33:08.6809603Z 2022-09-27T15:33:08.6809741Z Generating XML reports... 2022-09-27T15:33:08.6844902Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153306.xml 2022-09-27T15:33:10.0551598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:10.0552007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:10.0560579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:10.0561134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:10.2128514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptfub999d 2022-09-27T15:33:10.2129567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptfub999d/_remote_module_non_scriptable.py 2022-09-27T15:33:10.4753221Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:10.4763230Z 2022-09-27T15:33:10.4763376Z Running tests... 2022-09-27T15:33:10.4763976Z ---------------------------------------------------------------------- 2022-09-27T15:33:10.7201490Z test_backwards_nested_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11543 2022-09-27T15:33:10.7227862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11544 2022-09-27T15:33:10.7255817Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11545 2022-09-27T15:33:10.7284755Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11546 2022-09-27T15:33:11.8391116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:11.8391744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:11.8740185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:11.8740858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:11.8755395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:11.8756368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:11.8759884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:11.8760723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:11.9130483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:11.9131215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:11.9149991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:11.9150681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:11.9151546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:11.9152172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:11.9516972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:11.9517666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:12.0013212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6i97_c7d 2022-09-27T15:33:12.0014303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6i97_c7d/_remote_module_non_scriptable.py 2022-09-27T15:33:12.0432180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpee6cjrmc 2022-09-27T15:33:12.0432985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpltaed91w 2022-09-27T15:33:12.0433723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpee6cjrmc/_remote_module_non_scriptable.py 2022-09-27T15:33:12.0434637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpltaed91w/_remote_module_non_scriptable.py 2022-09-27T15:33:12.0795678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6__h1egl 2022-09-27T15:33:12.0796376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6__h1egl/_remote_module_non_scriptable.py 2022-09-27T15:33:12.2583644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:33:12.3012326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:33:12.3025257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:33:12.3377890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:13.0342618Z ok (2.558s) 2022-09-27T15:33:13.0342859Z 2022-09-27T15:33:13.0343295Z ---------------------------------------------------------------------- 2022-09-27T15:33:13.0343687Z Ran 1 test in 2.558s 2022-09-27T15:33:13.0343854Z 2022-09-27T15:33:13.0343952Z OK 2022-09-27T15:33:13.0344110Z 2022-09-27T15:33:13.0344276Z Generating XML reports... 2022-09-27T15:33:13.0380441Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153310.xml 2022-09-27T15:33:14.4355588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:14.4355969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:14.4364365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:14.4364835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:14.5959524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsulv_7oo 2022-09-27T15:33:14.5960443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsulv_7oo/_remote_module_non_scriptable.py 2022-09-27T15:33:14.8596942Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:14.8606147Z 2022-09-27T15:33:14.8606286Z Running tests... 2022-09-27T15:33:14.8607004Z ---------------------------------------------------------------------- 2022-09-27T15:33:14.8620797Z test_clean_context_during_backward (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:33:15.1104232Z This test simulates the situation where the 'backward' call might throw ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11786 2022-09-27T15:33:15.1131706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11787 2022-09-27T15:33:15.1159274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11788 2022-09-27T15:33:15.1187813Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11789 2022-09-27T15:33:16.2707658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:16.2708375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:16.2893439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:16.2894015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:16.3087166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:16.3087803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:16.3253413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:16.3254099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:16.3304693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:16.3305410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:16.3306220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:16.3306874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:16.3683901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:16.3684536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:16.3689298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:16.3689888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:16.4362243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgkkgz3hg 2022-09-27T15:33:16.4362975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgkkgz3hg/_remote_module_non_scriptable.py 2022-09-27T15:33:16.4520228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mlomn8u 2022-09-27T15:33:16.4521049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mlomn8u/_remote_module_non_scriptable.py 2022-09-27T15:33:16.4949204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjev1gg08 2022-09-27T15:33:16.4950028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjev1gg08/_remote_module_non_scriptable.py 2022-09-27T15:33:16.4956311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp04h_nbox 2022-09-27T15:33:16.4958351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp04h_nbox/_remote_module_non_scriptable.py 2022-09-27T15:33:16.6940504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:33:16.7066734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:33:16.7497661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:33:16.7518778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:16.9945640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:33:17.0049336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:33:17.0151819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:33:17.0152676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:33:17.0153933Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:17.0155192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:17.0156360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:17.0251183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:17.1883074Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:33:17.1889010Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:33:17.1889717Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:33:17.1933357Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:33:17.1934448Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:33:17.1935463Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:33:17.1957759Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:33:17.1958819Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:33:17.1959577Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:33:17.2159457Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:33:17.2160409Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:33:17.2161313Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:33:17.6249743Z ok (2.764s) 2022-09-27T15:33:17.6249956Z 2022-09-27T15:33:17.6250468Z ---------------------------------------------------------------------- 2022-09-27T15:33:17.6251119Z Ran 1 test in 2.764s 2022-09-27T15:33:17.6251238Z 2022-09-27T15:33:17.6251286Z OK 2022-09-27T15:33:17.6251377Z 2022-09-27T15:33:17.6251471Z Generating XML reports... 2022-09-27T15:33:17.6287357Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153314.xml 2022-09-27T15:33:19.0008018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:19.0008387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:19.0016219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:19.0016620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:19.1619211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe47hbt6f 2022-09-27T15:33:19.1620665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe47hbt6f/_remote_module_non_scriptable.py 2022-09-27T15:33:19.4228248Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:19.4238385Z 2022-09-27T15:33:19.4238478Z Running tests... 2022-09-27T15:33:19.4239183Z ---------------------------------------------------------------------- 2022-09-27T15:33:19.6654366Z test_context_cleanup_nested_rpc (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12025 2022-09-27T15:33:19.6682212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12026 2022-09-27T15:33:19.6708994Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12027 2022-09-27T15:33:19.6737243Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12028 2022-09-27T15:33:20.8095391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:20.8096051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:20.8326197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:20.8326877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:20.8366182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:20.8366849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:20.8459415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:20.8459935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:20.8717835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:20.8718482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:20.8751603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:20.8752222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:20.8796388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:20.8797031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:20.9176368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:20.9176984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:20.9746029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7k_wuy5z 2022-09-27T15:33:20.9746928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7k_wuy5z/_remote_module_non_scriptable.py 2022-09-27T15:33:21.0021139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvw56ei72 2022-09-27T15:33:21.0022022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvw56ei72/_remote_module_non_scriptable.py 2022-09-27T15:33:21.0118389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3mqtccr4 2022-09-27T15:33:21.0119688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3mqtccr4/_remote_module_non_scriptable.py 2022-09-27T15:33:21.0433925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphlpampuv 2022-09-27T15:33:21.0434818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphlpampuv/_remote_module_non_scriptable.py 2022-09-27T15:33:21.2286069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:33:21.2564647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:21.2670085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:33:21.2999729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:33:21.5344423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:33:21.5549966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:33:21.5652409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:33:21.5653915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:21.5656614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:21.5657790Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:21.5658558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:33:21.5664273Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:22.0797282Z ok (2.655s) 2022-09-27T15:33:22.0797520Z 2022-09-27T15:33:22.0798038Z ---------------------------------------------------------------------- 2022-09-27T15:33:22.0798359Z Ran 1 test in 2.656s 2022-09-27T15:33:22.0798476Z 2022-09-27T15:33:22.0798537Z OK 2022-09-27T15:33:22.0798630Z 2022-09-27T15:33:22.0798727Z Generating XML reports... 2022-09-27T15:33:22.0834630Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153319.xml 2022-09-27T15:33:23.4999349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:23.4999734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:23.5007487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:23.5007852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:23.6573650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxv819dhm 2022-09-27T15:33:23.6574422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxv819dhm/_remote_module_non_scriptable.py 2022-09-27T15:33:23.9181539Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:23.9190387Z 2022-09-27T15:33:23.9190475Z Running tests... 2022-09-27T15:33:23.9190990Z ---------------------------------------------------------------------- 2022-09-27T15:33:24.1635116Z test_context_cleanup_no_tensors (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12260 2022-09-27T15:33:24.1661582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12261 2022-09-27T15:33:24.1688854Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12262 2022-09-27T15:33:24.1716731Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12263 2022-09-27T15:33:25.2925827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:25.2926499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:25.3294008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:25.3294698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:25.3305410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:25.3306121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:25.3655729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:25.3656445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:25.4419052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:25.4419668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:25.4530281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:25.4531029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:25.4554780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2qzslru 2022-09-27T15:33:25.4556117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2qzslru/_remote_module_non_scriptable.py 2022-09-27T15:33:25.4802147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:25.4802862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:25.4915685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:25.4916277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:25.4917743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnu00siqz 2022-09-27T15:33:25.4920843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnu00siqz/_remote_module_non_scriptable.py 2022-09-27T15:33:25.6063613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqz50ia4 2022-09-27T15:33:25.6064861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqz50ia4/_remote_module_non_scriptable.py 2022-09-27T15:33:25.6179826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wyp72h4 2022-09-27T15:33:25.6183120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wyp72h4/_remote_module_non_scriptable.py 2022-09-27T15:33:25.7122709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:33:25.7479524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:33:25.8607940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:33:25.8754180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:26.1210598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:33:26.1309449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:33:26.1411316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:33:26.1412466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:33:26.1414102Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:26.1415537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:26.1416824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:26.1422283Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:26.6778047Z ok (2.758s) 2022-09-27T15:33:26.6778296Z 2022-09-27T15:33:26.6778784Z ---------------------------------------------------------------------- 2022-09-27T15:33:26.6779169Z Ran 1 test in 2.759s 2022-09-27T15:33:26.6779291Z 2022-09-27T15:33:26.6779352Z OK 2022-09-27T15:33:26.6779448Z 2022-09-27T15:33:26.6779542Z Generating XML reports... 2022-09-27T15:33:26.6814488Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153323.xml 2022-09-27T15:33:28.0530615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:28.0531030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:28.0539068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:28.0539432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:28.2124882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0drasgni 2022-09-27T15:33:28.2125833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0drasgni/_remote_module_non_scriptable.py 2022-09-27T15:33:28.4709212Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:28.4719415Z 2022-09-27T15:33:28.4719837Z Running tests... 2022-09-27T15:33:28.4720227Z ---------------------------------------------------------------------- 2022-09-27T15:33:28.7146924Z test_context_cleanup_tensor_no_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12495 2022-09-27T15:33:28.7173505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12496 2022-09-27T15:33:28.7200484Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12497 2022-09-27T15:33:28.7228781Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12498 2022-09-27T15:33:29.8333635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:29.8334270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:29.8671805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:29.8672459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:29.8692553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:29.8693136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:29.9048534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:29.9049186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:29.9121349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:29.9122280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:29.9352800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:29.9353750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:29.9486732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:29.9487259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:29.9713920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:29.9714649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:29.9962340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ndqqcug 2022-09-27T15:33:29.9963290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ndqqcug/_remote_module_non_scriptable.py 2022-09-27T15:33:30.0318140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwuue5az0 2022-09-27T15:33:30.0318922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwuue5az0/_remote_module_non_scriptable.py 2022-09-27T15:33:30.0749273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5935qh7y 2022-09-27T15:33:30.0750072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5935qh7y/_remote_module_non_scriptable.py 2022-09-27T15:33:30.0973250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5kuwyc0 2022-09-27T15:33:30.0974091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5kuwyc0/_remote_module_non_scriptable.py 2022-09-27T15:33:30.2564607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:33:30.3109734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:33:30.3348403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:33:30.3565251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:30.6128000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:33:30.6226372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:33:30.6330957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:33:30.6331825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:33:30.6333058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:30.6333975Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:30.6335004Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:30.6336094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:31.1287762Z ok (2.656s) 2022-09-27T15:33:31.1288033Z 2022-09-27T15:33:31.1288378Z ---------------------------------------------------------------------- 2022-09-27T15:33:31.1288656Z Ran 1 test in 2.657s 2022-09-27T15:33:31.1288775Z 2022-09-27T15:33:31.1288835Z OK 2022-09-27T15:33:31.1288915Z 2022-09-27T15:33:31.1289012Z Generating XML reports... 2022-09-27T15:33:31.1326663Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153328.xml 2022-09-27T15:33:32.5021097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:32.5021724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:32.5028856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:32.5029432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:32.6603900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9z7gz_jh 2022-09-27T15:33:32.6604778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9z7gz_jh/_remote_module_non_scriptable.py 2022-09-27T15:33:32.9213814Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:32.9222347Z 2022-09-27T15:33:32.9222478Z Running tests... 2022-09-27T15:33:32.9223051Z ---------------------------------------------------------------------- 2022-09-27T15:33:33.1652822Z test_context_cleanup_tensor_with_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12730 2022-09-27T15:33:33.1680025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12731 2022-09-27T15:33:33.1707169Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12732 2022-09-27T15:33:33.1734985Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12733 2022-09-27T15:33:34.2855884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:34.2856603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:34.2941292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:34.2941971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:34.2942780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:34.2943423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:34.3062897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:34.3063453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:34.3244925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:34.3245641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:34.3332464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:34.3333141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:34.3333939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:34.3334574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:34.3447080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:34.3447700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:34.4528128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98eogdya 2022-09-27T15:33:34.4529255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98eogdya/_remote_module_non_scriptable.py 2022-09-27T15:33:34.4612239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lky2_96 2022-09-27T15:33:34.4613955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lky2_96/_remote_module_non_scriptable.py 2022-09-27T15:33:34.4624544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeffva3d0 2022-09-27T15:33:34.4626767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeffva3d0/_remote_module_non_scriptable.py 2022-09-27T15:33:34.4868458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0nqtnggo 2022-09-27T15:33:34.4869369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0nqtnggo/_remote_module_non_scriptable.py 2022-09-27T15:33:34.7118073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:33:34.7185700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:33:34.7199323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:33:34.7468449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:35.0021293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:33:35.0126691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:33:35.0127331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:33:35.0130762Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:35.0131691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:33:35.0132799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:35.0224034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:35.0228401Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:35.5795213Z ok (2.657s) 2022-09-27T15:33:35.5795452Z 2022-09-27T15:33:35.5795950Z ---------------------------------------------------------------------- 2022-09-27T15:33:35.5796318Z Ran 1 test in 2.657s 2022-09-27T15:33:35.5796437Z 2022-09-27T15:33:35.5796498Z OK 2022-09-27T15:33:35.5796590Z 2022-09-27T15:33:35.5796692Z Generating XML reports... 2022-09-27T15:33:35.5831952Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153332.xml 2022-09-27T15:33:36.9541563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:36.9541933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:36.9549165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:36.9549531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:37.1111798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5nhxi9j 2022-09-27T15:33:37.1112674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5nhxi9j/_remote_module_non_scriptable.py 2022-09-27T15:33:37.3754340Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:37.3764830Z 2022-09-27T15:33:37.3765262Z Running tests... 2022-09-27T15:33:37.3765669Z ---------------------------------------------------------------------- 2022-09-27T15:33:37.6201065Z test_debug_info (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12965 2022-09-27T15:33:37.6228398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12966 2022-09-27T15:33:37.6256707Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12967 2022-09-27T15:33:37.6286419Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12968 2022-09-27T15:33:38.7528584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:38.7529530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:38.7888575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:38.7889571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:38.7938427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:38.7939080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:38.8114391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:38.8115249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:38.8302768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:38.8303336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:38.8432951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:38.8433686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:38.8495307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:38.8495932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:38.8817167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:38.8817832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:38.9169295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7zvchkt 2022-09-27T15:33:38.9170055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7zvchkt/_remote_module_non_scriptable.py 2022-09-27T15:33:38.9579023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpweb02_zz 2022-09-27T15:33:38.9579818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpweb02_zz/_remote_module_non_scriptable.py 2022-09-27T15:33:38.9774714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09gfdc2n 2022-09-27T15:33:38.9775883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09gfdc2n/_remote_module_non_scriptable.py 2022-09-27T15:33:39.0087371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdb0ej1d 2022-09-27T15:33:39.0088582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdb0ej1d/_remote_module_non_scriptable.py 2022-09-27T15:33:39.1741855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:33:39.2128918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:33:39.2354668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:39.2632998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:33:39.4969081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:33:39.5067368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:33:39.5168449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:33:39.5169256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:33:39.5171918Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:39.5173075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:39.5174494Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:39.5175892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:40.0347678Z ok (2.658s) 2022-09-27T15:33:40.0347941Z 2022-09-27T15:33:40.0348445Z ---------------------------------------------------------------------- 2022-09-27T15:33:40.0348804Z Ran 1 test in 2.658s 2022-09-27T15:33:40.0348909Z 2022-09-27T15:33:40.0349002Z OK 2022-09-27T15:33:40.0349115Z 2022-09-27T15:33:40.0349212Z Generating XML reports... 2022-09-27T15:33:40.0384128Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153337.xml 2022-09-27T15:33:41.3995813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:41.3996201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:41.4004553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:41.4004926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:41.5581324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpth1lmsff 2022-09-27T15:33:41.5582741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpth1lmsff/_remote_module_non_scriptable.py 2022-09-27T15:33:41.8206975Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:41.8217057Z 2022-09-27T15:33:41.8217187Z Running tests... 2022-09-27T15:33:41.8217653Z ---------------------------------------------------------------------- 2022-09-27T15:33:42.0601752Z test_dist_autograd_profiling (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77318 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.238s) 2022-09-27T15:33:42.0602254Z 2022-09-27T15:33:42.0602454Z ---------------------------------------------------------------------- 2022-09-27T15:33:42.0602712Z Ran 1 test in 0.238s 2022-09-27T15:33:42.0602829Z 2022-09-27T15:33:42.0602899Z OK (skipped=1) 2022-09-27T15:33:42.0603008Z 2022-09-27T15:33:42.0603093Z Generating XML reports... 2022-09-27T15:33:42.0625666Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153341.xml 2022-09-27T15:33:43.3685082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:43.3685757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:43.3695009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:43.3695553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:43.5312133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpascfrkw7 2022-09-27T15:33:43.5313025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpascfrkw7/_remote_module_non_scriptable.py 2022-09-27T15:33:43.7967201Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:43.7977626Z 2022-09-27T15:33:43.7978127Z Running tests... 2022-09-27T15:33:43.7978550Z ---------------------------------------------------------------------- 2022-09-27T15:33:44.0371194Z test_error_in_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13230 2022-09-27T15:33:44.0398871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13231 2022-09-27T15:33:44.0426076Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13232 2022-09-27T15:33:44.0453842Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13233 2022-09-27T15:33:45.1511054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:45.1511730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:45.1813563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:45.1814211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:45.1859028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:45.1859680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:45.2164332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:45.2164998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:45.2173936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:45.2174555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:45.2370043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:45.2370721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:45.2543641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:45.2544163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:45.2729876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:45.2730570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:45.3138141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gp2jjzs 2022-09-27T15:33:45.3138950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gp2jjzs/_remote_module_non_scriptable.py 2022-09-27T15:33:45.3437600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvzd99lnl 2022-09-27T15:33:45.3438560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvzd99lnl/_remote_module_non_scriptable.py 2022-09-27T15:33:45.3804381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ije0uc0 2022-09-27T15:33:45.3805147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ije0uc0/_remote_module_non_scriptable.py 2022-09-27T15:33:45.3987989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9en801r 2022-09-27T15:33:45.3988682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9en801r/_remote_module_non_scriptable.py 2022-09-27T15:33:45.5703265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:33:45.6012012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:33:45.6377842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:45.6532139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:33:46.3512032Z ok (2.553s) 2022-09-27T15:33:46.3512304Z 2022-09-27T15:33:46.3512833Z ---------------------------------------------------------------------- 2022-09-27T15:33:46.3513186Z Ran 1 test in 2.553s 2022-09-27T15:33:46.3513294Z 2022-09-27T15:33:46.3513358Z OK 2022-09-27T15:33:46.3513451Z 2022-09-27T15:33:46.3513547Z Generating XML reports... 2022-09-27T15:33:46.3548817Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153343.xml 2022-09-27T15:33:47.7305019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:47.7305648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:47.7313415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:47.7314071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:47.8897302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8xuwetj 2022-09-27T15:33:47.8897835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8xuwetj/_remote_module_non_scriptable.py 2022-09-27T15:33:48.1503251Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:48.1513631Z 2022-09-27T15:33:48.1513930Z Running tests... 2022-09-27T15:33:48.1514350Z ---------------------------------------------------------------------- 2022-09-27T15:33:48.3980365Z test_grad_copy_sparse_indices_extra_ref (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13453 2022-09-27T15:33:48.4007585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13454 2022-09-27T15:33:48.4034768Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13455 2022-09-27T15:33:48.4062950Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13456 2022-09-27T15:33:49.5067123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:49.5067791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:49.5356525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:49.5357182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:49.5449608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:49.5450236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:49.5527526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:49.5528191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:49.5552856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:49.5553381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:49.5750566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:49.5751180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:49.5916474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:49.5917147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:49.5941116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:49.5941723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:49.6735550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpbcxj4o 2022-09-27T15:33:49.6736333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpbcxj4o/_remote_module_non_scriptable.py 2022-09-27T15:33:49.7027392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs5chjo2 2022-09-27T15:33:49.7028439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs5chjo2/_remote_module_non_scriptable.py 2022-09-27T15:33:49.7184585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpls3ygq9r 2022-09-27T15:33:49.7185592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpls3ygq9r/_remote_module_non_scriptable.py 2022-09-27T15:33:49.7212424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ec9ty7t 2022-09-27T15:33:49.7214467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ec9ty7t/_remote_module_non_scriptable.py 2022-09-27T15:33:49.9352279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:49.9790802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:33:49.9796295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:33:49.9811972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:33:50.1860247Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-09-27T15:33:50.1861365Z "Argument order of nn.functional.embedding_bag was changed. " 2022-09-27T15:33:50.2057758Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-09-27T15:33:50.2061663Z "Argument order of nn.functional.embedding_bag was changed. " 2022-09-27T15:33:50.2063509Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-09-27T15:33:50.2064990Z "Argument order of nn.functional.embedding_bag was changed. " 2022-09-27T15:33:50.2066704Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-09-27T15:33:50.2068160Z "Argument order of nn.functional.embedding_bag was changed. " 2022-09-27T15:33:50.6163378Z ok (2.465s) 2022-09-27T15:33:50.6163617Z 2022-09-27T15:33:50.6163993Z ---------------------------------------------------------------------- 2022-09-27T15:33:50.6164233Z Ran 1 test in 2.465s 2022-09-27T15:33:50.6164353Z 2022-09-27T15:33:50.6164417Z OK 2022-09-27T15:33:50.6164509Z 2022-09-27T15:33:50.6164606Z Generating XML reports... 2022-09-27T15:33:50.6199814Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153348.xml 2022-09-27T15:33:52.0059660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:52.0060057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:52.0067859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:52.0068221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:52.1680533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxz2stp61 2022-09-27T15:33:52.1681990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxz2stp61/_remote_module_non_scriptable.py 2022-09-27T15:33:52.4302969Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:52.4312482Z 2022-09-27T15:33:52.4312585Z Running tests... 2022-09-27T15:33:52.4313181Z ---------------------------------------------------------------------- 2022-09-27T15:33:52.6722696Z test_grad_only_on_return_value (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13708 2022-09-27T15:33:52.6749172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13709 2022-09-27T15:33:52.6776503Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13710 2022-09-27T15:33:52.6804816Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13711 2022-09-27T15:33:53.8442947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:53.8443600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:53.8775992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:53.8776626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:53.8864860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:53.8865484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:53.9180335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:53.9181009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:53.9241403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:53.9241945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:53.9396241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:53.9396804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:53.9572450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:53.9573133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:53.9775484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:53.9776171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:54.0227613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1v76_t61 2022-09-27T15:33:54.0228382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1v76_t61/_remote_module_non_scriptable.py 2022-09-27T15:33:54.0524890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbp52nf8x 2022-09-27T15:33:54.0525755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbp52nf8x/_remote_module_non_scriptable.py 2022-09-27T15:33:54.0837349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1lq4aee 2022-09-27T15:33:54.0838177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1lq4aee/_remote_module_non_scriptable.py 2022-09-27T15:33:54.1057591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_z8twozh 2022-09-27T15:33:54.1058390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_z8twozh/_remote_module_non_scriptable.py 2022-09-27T15:33:54.2807616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:33:54.3083476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:33:54.3380595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:54.3604693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:33:54.6210262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:33:54.6210855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:33:54.6311279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:33:54.6312515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:33:54.6314812Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:54.6315943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:54.6316765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:54.6318322Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:55.1865551Z ok (2.755s) 2022-09-27T15:33:55.1865849Z 2022-09-27T15:33:55.1866347Z ---------------------------------------------------------------------- 2022-09-27T15:33:55.1866628Z Ran 1 test in 2.755s 2022-09-27T15:33:55.1866749Z 2022-09-27T15:33:55.1866809Z OK 2022-09-27T15:33:55.1866899Z 2022-09-27T15:33:55.1866995Z Generating XML reports... 2022-09-27T15:33:55.1902461Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153352.xml 2022-09-27T15:33:56.5661022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:56.5661403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:56.5669199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:56.5669565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:56.7246403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_i0_qu24 2022-09-27T15:33:56.7247216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_i0_qu24/_remote_module_non_scriptable.py 2022-09-27T15:33:56.9846387Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:33:56.9855459Z 2022-09-27T15:33:56.9855602Z Running tests... 2022-09-27T15:33:56.9856339Z ---------------------------------------------------------------------- 2022-09-27T15:33:57.2298338Z test_grad_only_on_return_value_remote (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13963 2022-09-27T15:33:57.2325964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13964 2022-09-27T15:33:57.2353261Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13965 2022-09-27T15:33:57.2382731Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13966 2022-09-27T15:33:58.3777210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:58.3777906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:58.3795052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:58.3795755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:58.3839593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:58.3840254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:58.4162418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:58.4163231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:58.4180511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:58.4181151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:58.4217124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:58.4217741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:58.4406736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:33:58.4407389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:33:58.4759877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:33:58.4760516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:33:58.5427286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgiq16psb 2022-09-27T15:33:58.5428096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgiq16psb/_remote_module_non_scriptable.py 2022-09-27T15:33:58.5460559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lwhen4k 2022-09-27T15:33:58.5462567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lwhen4k/_remote_module_non_scriptable.py 2022-09-27T15:33:58.5476912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23bf5no1 2022-09-27T15:33:58.5479815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23bf5no1/_remote_module_non_scriptable.py 2022-09-27T15:33:58.6028031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqb9_v9g 2022-09-27T15:33:58.6029374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqb9_v9g/_remote_module_non_scriptable.py 2022-09-27T15:33:58.7974172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:33:58.8038659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:33:58.8048549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:33:58.8596288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:33:59.1230646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:33:59.1231552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:33:59.1343519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:33:59.1344070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:33:59.1344890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:59.1345657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:59.1346416Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:59.1350571Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:33:59.7442948Z ok (2.758s) 2022-09-27T15:33:59.7443251Z 2022-09-27T15:33:59.7443788Z ---------------------------------------------------------------------- 2022-09-27T15:33:59.7444114Z Ran 1 test in 2.759s 2022-09-27T15:33:59.7444233Z 2022-09-27T15:33:59.7444295Z OK 2022-09-27T15:33:59.7444374Z 2022-09-27T15:33:59.7444466Z Generating XML reports... 2022-09-27T15:33:59.7479240Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153356.xml 2022-09-27T15:34:01.1001983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:01.1002469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:01.1010849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:01.1011212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:01.2661895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy14e47uw 2022-09-27T15:34:01.2662672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy14e47uw/_remote_module_non_scriptable.py 2022-09-27T15:34:01.5347078Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:34:01.5357362Z 2022-09-27T15:34:01.5357483Z Running tests... 2022-09-27T15:34:01.5357880Z ---------------------------------------------------------------------- 2022-09-27T15:34:01.7835126Z test_graph_for_builtin_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14218 2022-09-27T15:34:01.7861983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14219 2022-09-27T15:34:01.7889816Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14220 2022-09-27T15:34:01.7919295Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14221 2022-09-27T15:34:02.9339993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:02.9340606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:02.9353429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:02.9354096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:02.9702790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:02.9703529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:02.9716378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:02.9717008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:02.9764656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:02.9765289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:03.0061117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:03.0061754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:03.0125380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:03.0125955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:03.0423542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:03.0424175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:03.0976591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsva3btex 2022-09-27T15:34:03.0977441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsva3btex/_remote_module_non_scriptable.py 2022-09-27T15:34:03.0984420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7s4luoxo 2022-09-27T15:34:03.0986825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7s4luoxo/_remote_module_non_scriptable.py 2022-09-27T15:34:03.1384525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcqsgikkr 2022-09-27T15:34:03.1385344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcqsgikkr/_remote_module_non_scriptable.py 2022-09-27T15:34:03.1687885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzc3z10lh 2022-09-27T15:34:03.1688612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzc3z10lh/_remote_module_non_scriptable.py 2022-09-27T15:34:03.3542033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:34:03.3719803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:34:03.3933008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:34:03.4308490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:34:03.6767155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:34:03.6867796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:34:03.6968530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:34:03.6973600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:34:03.6974829Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:03.7073976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:03.7075574Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:03.7077840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:04.2980669Z ok (2.762s) 2022-09-27T15:34:04.2980940Z 2022-09-27T15:34:04.2981445Z ---------------------------------------------------------------------- 2022-09-27T15:34:04.2981756Z Ran 1 test in 2.762s 2022-09-27T15:34:04.2981873Z 2022-09-27T15:34:04.2981932Z OK 2022-09-27T15:34:04.2982024Z 2022-09-27T15:34:04.2982107Z Generating XML reports... 2022-09-27T15:34:04.3016539Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153401.xml 2022-09-27T15:34:05.6657923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:05.6658303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:05.6666035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:05.6666387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:05.8216045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpconbnjl6 2022-09-27T15:34:05.8217182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpconbnjl6/_remote_module_non_scriptable.py 2022-09-27T15:34:06.0851026Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:34:06.0860651Z 2022-09-27T15:34:06.0861040Z Running tests... 2022-09-27T15:34:06.0861426Z ---------------------------------------------------------------------- 2022-09-27T15:34:06.3285962Z test_graph_for_builtin_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14453 2022-09-27T15:34:06.3313816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14454 2022-09-27T15:34:06.3341994Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14455 2022-09-27T15:34:06.3370200Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14456 2022-09-27T15:34:07.4527991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:07.4528746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:07.4719818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:07.4720470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:07.4800753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:07.4801436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:07.4903159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:07.4903735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:07.5096200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:07.5096870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:07.5170831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:07.5171488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:07.5250257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:07.5250879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:07.5614600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:07.5615289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:07.6175631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp98604mt 2022-09-27T15:34:07.6176423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp98604mt/_remote_module_non_scriptable.py 2022-09-27T15:34:07.6355286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5awx_sh 2022-09-27T15:34:07.6356097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5awx_sh/_remote_module_non_scriptable.py 2022-09-27T15:34:07.6438960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmc6jplc 2022-09-27T15:34:07.6440197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmc6jplc/_remote_module_non_scriptable.py 2022-09-27T15:34:07.6875906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppj5qpvjk 2022-09-27T15:34:07.6876879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppj5qpvjk/_remote_module_non_scriptable.py 2022-09-27T15:34:07.8771488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:34:07.8940709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:34:07.9036631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:34:07.9457376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:34:08.1848732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:34:08.1949438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:34:08.2050705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:34:08.2055517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:34:08.2056790Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:08.2156654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:08.2157684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:08.2158477Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:08.7430461Z ok (2.657s) 2022-09-27T15:34:08.7430738Z 2022-09-27T15:34:08.7431183Z ---------------------------------------------------------------------- 2022-09-27T15:34:08.7431525Z Ran 1 test in 2.657s 2022-09-27T15:34:08.7431634Z 2022-09-27T15:34:08.7431696Z OK 2022-09-27T15:34:08.7431794Z 2022-09-27T15:34:08.7431890Z Generating XML reports... 2022-09-27T15:34:08.7467071Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153406.xml 2022-09-27T15:34:10.1134253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:10.1134637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:10.1142467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:10.1143007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:10.2709188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptuk6bpi1 2022-09-27T15:34:10.2709953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptuk6bpi1/_remote_module_non_scriptable.py 2022-09-27T15:34:10.5304113Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:34:10.5313582Z 2022-09-27T15:34:10.5313704Z Running tests... 2022-09-27T15:34:10.5314188Z ---------------------------------------------------------------------- 2022-09-27T15:34:10.7751234Z test_graph_for_py_nested_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14688 2022-09-27T15:34:10.7778884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14689 2022-09-27T15:34:10.7805657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14690 2022-09-27T15:34:10.7833698Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14691 2022-09-27T15:34:11.8840311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:11.8840985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:11.8850539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:11.8851182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:11.8871631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:11.8872280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:11.9164072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:11.9164774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:11.9232343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:11.9232962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:11.9240833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:11.9241409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:11.9259466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:11.9260108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:11.9542611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:11.9543262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:12.0512945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bnm_jgf 2022-09-27T15:34:12.0513648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5njuooj 2022-09-27T15:34:12.0514316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bnm_jgf/_remote_module_non_scriptable.py 2022-09-27T15:34:12.0515117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5njuooj/_remote_module_non_scriptable.py 2022-09-27T15:34:12.0517585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaj24tj2q 2022-09-27T15:34:12.0520378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaj24tj2q/_remote_module_non_scriptable.py 2022-09-27T15:34:12.0811351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51a9q41t 2022-09-27T15:34:12.0812085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51a9q41t/_remote_module_non_scriptable.py 2022-09-27T15:34:12.3069082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:34:12.3069995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:34:12.3094628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:34:12.3421971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:34:12.6127467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:34:12.6229834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:34:12.6330848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:34:12.6335834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:34:12.6337091Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:12.6338252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:12.6434970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:12.6437198Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:13.1893392Z ok (2.658s) 2022-09-27T15:34:13.1893715Z 2022-09-27T15:34:13.1894227Z ---------------------------------------------------------------------- 2022-09-27T15:34:13.1894587Z Ran 1 test in 2.658s 2022-09-27T15:34:13.1894704Z 2022-09-27T15:34:13.1894763Z OK 2022-09-27T15:34:13.1894842Z 2022-09-27T15:34:13.1894936Z Generating XML reports... 2022-09-27T15:34:13.1932845Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153410.xml 2022-09-27T15:34:14.5539239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:14.5539896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:14.5548588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:14.5549231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:14.7085023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tb2qb58 2022-09-27T15:34:14.7085703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tb2qb58/_remote_module_non_scriptable.py 2022-09-27T15:34:14.9688270Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:34:14.9698635Z 2022-09-27T15:34:14.9699060Z Running tests... 2022-09-27T15:34:14.9699491Z ---------------------------------------------------------------------- 2022-09-27T15:34:15.2146310Z test_graph_for_py_nested_call_itself (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14923 2022-09-27T15:34:15.2174004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14924 2022-09-27T15:34:15.2201490Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14925 2022-09-27T15:34:15.2230368Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14926 2022-09-27T15:34:16.2964909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:16.3005425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:16.3254213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:16.3254878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:16.3330982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:16.3331483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:16.3430957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:16.3431703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:16.3483907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:16.3484554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:16.3644669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:16.3645229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:16.3816795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:16.3817403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:16.3874214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:16.3874996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:16.4611284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpna6fxtqg 2022-09-27T15:34:16.4611954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpna6fxtqg/_remote_module_non_scriptable.py 2022-09-27T15:34:16.4918349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzb62rfi5 2022-09-27T15:34:16.4919328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzb62rfi5/_remote_module_non_scriptable.py 2022-09-27T15:34:16.5090873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu784s4w6 2022-09-27T15:34:16.5092059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu784s4w6/_remote_module_non_scriptable.py 2022-09-27T15:34:16.5153108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppxd8haxj 2022-09-27T15:34:16.5154399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppxd8haxj/_remote_module_non_scriptable.py 2022-09-27T15:34:16.7164653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:34:16.7509836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:34:16.7675842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:34:16.7771125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:34:17.0474862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:34:17.0574847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:34:17.0683982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:34:17.0685298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:17.0686449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:17.0687586Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:17.0688426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:34:17.0689479Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:17.6290938Z ok (2.659s) 2022-09-27T15:34:17.6291189Z 2022-09-27T15:34:17.6291730Z ---------------------------------------------------------------------- 2022-09-27T15:34:17.6292052Z Ran 1 test in 2.659s 2022-09-27T15:34:17.6292170Z 2022-09-27T15:34:17.6292230Z OK 2022-09-27T15:34:17.6292322Z 2022-09-27T15:34:17.6292425Z Generating XML reports... 2022-09-27T15:34:17.6328581Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153414.xml 2022-09-27T15:34:18.9942781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:18.9943235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:18.9950890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:18.9951236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:19.1504879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgsvqd96 2022-09-27T15:34:19.1506007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgsvqd96/_remote_module_non_scriptable.py 2022-09-27T15:34:19.4100109Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:34:19.4109764Z 2022-09-27T15:34:19.4109894Z Running tests... 2022-09-27T15:34:19.4110354Z ---------------------------------------------------------------------- 2022-09-27T15:34:19.6511121Z test_graph_for_py_nested_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15158 2022-09-27T15:34:19.6538532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15159 2022-09-27T15:34:19.6566091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15160 2022-09-27T15:34:19.6594185Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15161 2022-09-27T15:34:20.8083047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:20.8083690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:20.8448788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:20.8449423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:20.8654391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:20.8655145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:20.8660299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:20.8660950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:20.8783176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:20.8783696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:20.9048737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:20.9049421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:20.9057298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:20.9057893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:20.9174401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:20.9175074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:20.9903060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzsnddbbp 2022-09-27T15:34:20.9903886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzsnddbbp/_remote_module_non_scriptable.py 2022-09-27T15:34:21.0309278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzl2z1b46 2022-09-27T15:34:21.0310017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzl2z1b46/_remote_module_non_scriptable.py 2022-09-27T15:34:21.0327079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ep1f45b 2022-09-27T15:34:21.0329323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ep1f45b/_remote_module_non_scriptable.py 2022-09-27T15:34:21.0433828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvl7fhivt 2022-09-27T15:34:21.0436166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvl7fhivt/_remote_module_non_scriptable.py 2022-09-27T15:34:21.2490593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:34:21.2878288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:34:21.2908859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:34:21.3008474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:34:21.5584462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:34:21.5686268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:34:21.5787642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:34:21.5791285Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:21.5792545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:21.5793546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:34:21.5794449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:21.5795458Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:22.1656736Z ok (2.754s) 2022-09-27T15:34:22.1657319Z 2022-09-27T15:34:22.1657837Z ---------------------------------------------------------------------- 2022-09-27T15:34:22.1658178Z Ran 1 test in 2.755s 2022-09-27T15:34:22.1658295Z 2022-09-27T15:34:22.1658362Z OK 2022-09-27T15:34:22.1658441Z 2022-09-27T15:34:22.1658541Z Generating XML reports... 2022-09-27T15:34:22.1693647Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153419.xml 2022-09-27T15:34:23.5453901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:23.5454276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:23.5461865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:23.5462212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:23.7043354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmpxcx3q9 2022-09-27T15:34:23.7044508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmpxcx3q9/_remote_module_non_scriptable.py 2022-09-27T15:34:23.9694840Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:34:23.9703549Z 2022-09-27T15:34:23.9703681Z Running tests... 2022-09-27T15:34:23.9704049Z ---------------------------------------------------------------------- 2022-09-27T15:34:24.2159937Z test_graph_for_py_nested_remote_call_itself (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15393 2022-09-27T15:34:24.2187654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15394 2022-09-27T15:34:24.2216955Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15395 2022-09-27T15:34:24.2246680Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15396 2022-09-27T15:34:25.3687173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:25.3687864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:25.4043099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:25.4043713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:25.4079165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:25.4155128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:25.4156038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:25.4156668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:25.4274964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:25.4275450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:25.4453356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:25.4454013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:25.4529399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:25.4530025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:25.4652533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:25.4653180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:25.5313320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03ds50wk 2022-09-27T15:34:25.5314040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03ds50wk/_remote_module_non_scriptable.py 2022-09-27T15:34:25.5722363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tpw__dn 2022-09-27T15:34:25.5723013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tpw__dn/_remote_module_non_scriptable.py 2022-09-27T15:34:25.5798289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6xs2wss 2022-09-27T15:34:25.5799767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6xs2wss/_remote_module_non_scriptable.py 2022-09-27T15:34:25.5935498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplo438iaq 2022-09-27T15:34:25.5936852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplo438iaq/_remote_module_non_scriptable.py 2022-09-27T15:34:25.7919882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:34:25.8308012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:34:25.8420537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:34:25.8560897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:34:26.1412963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:34:26.1512617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:34:26.1616455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:34:26.1617322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:26.1617893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:34:26.1621251Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:26.1622855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:26.1624228Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:26.7308713Z ok (2.760s) 2022-09-27T15:34:26.7309037Z 2022-09-27T15:34:26.7309514Z ---------------------------------------------------------------------- 2022-09-27T15:34:26.7309837Z Ran 1 test in 2.760s 2022-09-27T15:34:26.7309952Z 2022-09-27T15:34:26.7310014Z OK 2022-09-27T15:34:26.7310093Z 2022-09-27T15:34:26.7310187Z Generating XML reports... 2022-09-27T15:34:26.7344909Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153423.xml 2022-09-27T15:34:28.1168700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:28.1169080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:28.1177016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:28.1177523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:28.2740714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhscmh96 2022-09-27T15:34:28.2741857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhscmh96/_remote_module_non_scriptable.py 2022-09-27T15:34:28.5383812Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:34:28.5393405Z 2022-09-27T15:34:28.5393544Z Running tests... 2022-09-27T15:34:28.5394224Z ---------------------------------------------------------------------- 2022-09-27T15:34:28.7830082Z test_graph_for_python_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15628 2022-09-27T15:34:28.7857658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15629 2022-09-27T15:34:28.7885542Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15630 2022-09-27T15:34:28.7914399Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15631 2022-09-27T15:34:29.9177258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:29.9177927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:29.9422848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:29.9423529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:29.9459163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:29.9459835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:29.9547985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:29.9548493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:29.9690640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:29.9691277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:29.9807815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:29.9808371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:29.9843978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:29.9844534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:30.0056755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:30.0057347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:30.0821560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56492pqy 2022-09-27T15:34:30.0822323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56492pqy/_remote_module_non_scriptable.py 2022-09-27T15:34:30.1071002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpipjkvpvx 2022-09-27T15:34:30.1072456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpipjkvpvx/_remote_module_non_scriptable.py 2022-09-27T15:34:30.1106756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytasecrn 2022-09-27T15:34:30.1108096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytasecrn/_remote_module_non_scriptable.py 2022-09-27T15:34:30.1342537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_d2zfbd 2022-09-27T15:34:30.1343395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_d2zfbd/_remote_module_non_scriptable.py 2022-09-27T15:34:30.3426959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:34:30.3686335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:34:30.3744875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:34:30.4155334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:34:30.6843811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:34:30.6947917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:34:30.7045993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:34:30.7047365Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:30.7048360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:34:30.7051607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:30.7052598Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:30.7054177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:31.2975860Z ok (2.758s) 2022-09-27T15:34:31.2976115Z 2022-09-27T15:34:31.2976422Z ---------------------------------------------------------------------- 2022-09-27T15:34:31.2976679Z Ran 1 test in 2.758s 2022-09-27T15:34:31.2976798Z 2022-09-27T15:34:31.2976859Z OK 2022-09-27T15:34:31.2976951Z 2022-09-27T15:34:31.2977033Z Generating XML reports... 2022-09-27T15:34:31.3013096Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153428.xml 2022-09-27T15:34:32.6908628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:32.6909290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:32.6918936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:32.6919569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:32.8509059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqe6ayz2o 2022-09-27T15:34:32.8509785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqe6ayz2o/_remote_module_non_scriptable.py 2022-09-27T15:34:33.1141350Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:34:33.1152170Z 2022-09-27T15:34:33.1152695Z Running tests... 2022-09-27T15:34:33.1153130Z ---------------------------------------------------------------------- 2022-09-27T15:34:33.3579080Z test_graph_for_python_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15863 2022-09-27T15:34:33.3605791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15864 2022-09-27T15:34:33.3633240Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15865 2022-09-27T15:34:33.3661918Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15866 2022-09-27T15:34:34.4440007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:34.4440672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:34.4612368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:34.4613015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:34.4817931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:34.4818498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:34.4894058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:34.4894688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:34.4995426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:34.4996076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:34.5158075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:34.5158756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:34.5266470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:34.5267018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:34.5524323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:34.5524940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:34.6094582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeo5l4_jx 2022-09-27T15:34:34.6095527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeo5l4_jx/_remote_module_non_scriptable.py 2022-09-27T15:34:34.6264570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp169nf41f 2022-09-27T15:34:34.6265346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp169nf41f/_remote_module_non_scriptable.py 2022-09-27T15:34:34.6525147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpta6y5ui3 2022-09-27T15:34:34.6525979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpta6y5ui3/_remote_module_non_scriptable.py 2022-09-27T15:34:34.6784771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpst2tl1_4 2022-09-27T15:34:34.6785759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpst2tl1_4/_remote_module_non_scriptable.py 2022-09-27T15:34:34.8655635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:34:34.8846324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:34:34.9102129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:34:34.9371635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:34:35.2290647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:34:35.2391254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:34:35.2394460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:34:35.2395456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:34:35.2396754Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:35.2397917Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:35.2399039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:35.2400159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:35.8724856Z ok (2.757s) 2022-09-27T15:34:35.8725101Z 2022-09-27T15:34:35.8725411Z ---------------------------------------------------------------------- 2022-09-27T15:34:35.8725665Z Ran 1 test in 2.757s 2022-09-27T15:34:35.8725782Z 2022-09-27T15:34:35.8725844Z OK 2022-09-27T15:34:35.8725937Z 2022-09-27T15:34:35.8726018Z Generating XML reports... 2022-09-27T15:34:35.8760847Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153433.xml 2022-09-27T15:34:37.2384052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:37.2384509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:37.2391514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:37.2391862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:37.3974912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85b9pgkn 2022-09-27T15:34:37.3975597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85b9pgkn/_remote_module_non_scriptable.py 2022-09-27T15:34:37.6583755Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:34:37.6593065Z 2022-09-27T15:34:37.6593186Z Running tests... 2022-09-27T15:34:37.6593820Z ---------------------------------------------------------------------- 2022-09-27T15:34:37.9012988Z test_mixed_requires_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16098 2022-09-27T15:34:37.9041552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16099 2022-09-27T15:34:37.9069889Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16100 2022-09-27T15:34:37.9099168Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16101 2022-09-27T15:34:38.9938117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:38.9938941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:38.9988059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:38.9988586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:39.0136067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:39.0136570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:39.0234068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:39.0234787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:39.0333128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:39.0333678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:39.0385189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:39.0385736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:39.0526275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:39.0526951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:39.0617884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:39.0618561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:39.1615461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoqjm0pz1 2022-09-27T15:34:39.1616260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoqjm0pz1/_remote_module_non_scriptable.py 2022-09-27T15:34:39.1653458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3b966e5 2022-09-27T15:34:39.1655001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3b966e5/_remote_module_non_scriptable.py 2022-09-27T15:34:39.1790767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbtvckwd 2022-09-27T15:34:39.1791962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbtvckwd/_remote_module_non_scriptable.py 2022-09-27T15:34:39.1882364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnd4y81i 2022-09-27T15:34:39.1883417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnd4y81i/_remote_module_non_scriptable.py 2022-09-27T15:34:39.4185527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:34:39.4217477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:34:39.4340853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:34:39.4439621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:34:40.1156216Z ok (2.456s) 2022-09-27T15:34:40.1156518Z 2022-09-27T15:34:40.1156996Z ---------------------------------------------------------------------- 2022-09-27T15:34:40.1157261Z Ran 1 test in 2.456s 2022-09-27T15:34:40.1157377Z 2022-09-27T15:34:40.1157439Z OK 2022-09-27T15:34:40.1157524Z 2022-09-27T15:34:40.1157619Z Generating XML reports... 2022-09-27T15:34:40.1194383Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153437.xml 2022-09-27T15:34:41.4984655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:41.4985319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:41.4994110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:41.4994964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:41.6585409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptumhnvvx 2022-09-27T15:34:41.6586088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptumhnvvx/_remote_module_non_scriptable.py 2022-09-27T15:34:41.9176481Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:34:41.9186598Z 2022-09-27T15:34:41.9187100Z Running tests... 2022-09-27T15:34:41.9187518Z ---------------------------------------------------------------------- 2022-09-27T15:34:42.1633200Z test_multiple_backward (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16341 2022-09-27T15:34:42.1659908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16342 2022-09-27T15:34:42.1687614Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16343 2022-09-27T15:34:42.1715737Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16344 2022-09-27T15:34:43.2524431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:43.2525082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:43.2703628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:43.2704352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:43.2904238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:43.2904779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:43.3078845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:43.3079546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:43.3157157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:43.3158069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:43.3194134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:43.3195305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:43.3539062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:43.3539683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:43.3578026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:43.3578638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:43.4187111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28hbnmqk 2022-09-27T15:34:43.4187943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28hbnmqk/_remote_module_non_scriptable.py 2022-09-27T15:34:43.4364094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp181ie_l5 2022-09-27T15:34:43.4364929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp181ie_l5/_remote_module_non_scriptable.py 2022-09-27T15:34:43.4821447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqa2v8na5 2022-09-27T15:34:43.4822243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqa2v8na5/_remote_module_non_scriptable.py 2022-09-27T15:34:43.4844570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphlsfoit_ 2022-09-27T15:34:43.4846140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphlsfoit_/_remote_module_non_scriptable.py 2022-09-27T15:34:43.6786704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:34:43.6999994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:34:43.7465255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:34:43.7649132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:34:45.1786713Z ok (3.260s) 2022-09-27T15:34:45.1787032Z 2022-09-27T15:34:45.1787408Z ---------------------------------------------------------------------- 2022-09-27T15:34:45.1787659Z Ran 1 test in 3.260s 2022-09-27T15:34:45.1787786Z 2022-09-27T15:34:45.1787848Z OK 2022-09-27T15:34:45.1787943Z 2022-09-27T15:34:45.1788023Z Generating XML reports... 2022-09-27T15:34:45.1822698Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153441.xml 2022-09-27T15:34:46.5518887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:46.5519257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:46.5526153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:46.5526702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:46.7080901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gixv12b 2022-09-27T15:34:46.7081961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gixv12b/_remote_module_non_scriptable.py 2022-09-27T15:34:46.9801417Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:34:46.9810831Z 2022-09-27T15:34:46.9810939Z Running tests... 2022-09-27T15:34:46.9811939Z ---------------------------------------------------------------------- 2022-09-27T15:34:47.2202454Z test_multiple_backward_with_errors (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16584 2022-09-27T15:34:47.2230153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16585 2022-09-27T15:34:47.2257279Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16586 2022-09-27T15:34:47.2284487Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16587 2022-09-27T15:34:48.3004705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:48.3005400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:48.3210689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:48.3211316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:48.3289243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:48.3289843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:48.3393020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:48.3393567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:48.3417983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:48.3418490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:48.3596178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:48.3596795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:48.3670690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:48.3671346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:48.3780915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:48.3781539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:48.4657999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckz3j3jp 2022-09-27T15:34:48.4658770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckz3j3jp/_remote_module_non_scriptable.py 2022-09-27T15:34:48.4855990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp493kh591 2022-09-27T15:34:48.4856989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp493kh591/_remote_module_non_scriptable.py 2022-09-27T15:34:48.4940740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjpkh3_8j 2022-09-27T15:34:48.4942307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjpkh3_8j/_remote_module_non_scriptable.py 2022-09-27T15:34:48.5034415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmug73gwy 2022-09-27T15:34:48.5036482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmug73gwy/_remote_module_non_scriptable.py 2022-09-27T15:34:48.7207142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:34:48.7417453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:34:48.7503975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:34:48.7615795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:34:49.0129546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:34:49.0328921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:34:49.0330125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:34:49.0334798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:34:49.0336175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:49.0336926Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:49.0436002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:49.0437110Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:34:49.2917062Z [W tensorpipe_agent.cpp:625] RPC agent for worker0 won't send response to request #150 to worker1, as the agent is shutting down 2022-09-27T15:34:49.2923232Z [W tensorpipe_agent.cpp:940] RPC agent for worker3 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:34:49.2924373Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:34:49.2925720Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:34:49.2926958Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:34:49.2928208Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:34:49.2935689Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:34:49.2936694Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:34:49.2937572Z [E container.cpp:256] Could not release Dist Autograd Context on node 0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:34:49.2938382Z [E container.cpp:256] Could not release Dist Autograd Context on node 2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:34:49.2961147Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:34:49.2962108Z [E container.cpp:256] Could not release Dist Autograd Context on node 0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:34:49.2963042Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:34:49.2981867Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #150 to worker2, as the agent is shutting down 2022-09-27T15:34:49.2984113Z [W tensorpipe_agent.cpp:625] RPC agent for worker0 won't send response to request #150 to worker3, as the agent is shutting down 2022-09-27T15:34:49.3032687Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:34:49.3158214Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:34:49.3158958Z [E container.cpp:256] Could not release Dist Autograd Context on node 3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:34:49.7347315Z ok (2.753s) 2022-09-27T15:34:49.7347521Z 2022-09-27T15:34:49.7347958Z ---------------------------------------------------------------------- 2022-09-27T15:34:49.7348332Z Ran 1 test in 2.753s 2022-09-27T15:34:49.7348540Z 2022-09-27T15:34:49.7348641Z OK 2022-09-27T15:34:49.7348786Z 2022-09-27T15:34:49.7348930Z Generating XML reports... 2022-09-27T15:34:49.7384708Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153446.xml 2022-09-27T15:34:51.0979778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:51.0980146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:51.0989016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:51.0989384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:51.2604950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9trivsc 2022-09-27T15:34:51.2606237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9trivsc/_remote_module_non_scriptable.py 2022-09-27T15:34:51.5208537Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:34:51.5218028Z 2022-09-27T15:34:51.5218266Z Running tests... 2022-09-27T15:34:51.5218678Z ---------------------------------------------------------------------- 2022-09-27T15:34:51.7634413Z test_nested_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16839 2022-09-27T15:34:51.7661706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16840 2022-09-27T15:34:51.7689179Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16841 2022-09-27T15:34:51.7717377Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16842 2022-09-27T15:34:52.9094087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:52.9094760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:52.9095573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:52.9096204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:52.9374781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:52.9375458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:52.9469907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:52.9470474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:52.9473043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:52.9473553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:52.9685859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:52.9686624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:52.9845736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:52.9846131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:53.0056443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:53.0057150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:53.0749586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe947koq3 2022-09-27T15:34:53.0750410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe947koq3/_remote_module_non_scriptable.py 2022-09-27T15:34:53.0757688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_9bzsv99 2022-09-27T15:34:53.0759665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_9bzsv99/_remote_module_non_scriptable.py 2022-09-27T15:34:53.1152983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8z344_af 2022-09-27T15:34:53.1154639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8z344_af/_remote_module_non_scriptable.py 2022-09-27T15:34:53.1313064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrnlzwkg 2022-09-27T15:34:53.1313981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrnlzwkg/_remote_module_non_scriptable.py 2022-09-27T15:34:53.3275638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:34:53.3311690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:34:53.3732126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:34:53.3845887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:34:54.0774938Z ok (2.555s) 2022-09-27T15:34:54.0775187Z 2022-09-27T15:34:54.0775686Z ---------------------------------------------------------------------- 2022-09-27T15:34:54.0776125Z Ran 1 test in 2.556s 2022-09-27T15:34:54.0776339Z 2022-09-27T15:34:54.0776452Z OK 2022-09-27T15:34:54.0776623Z 2022-09-27T15:34:54.0776799Z Generating XML reports... 2022-09-27T15:34:54.0812194Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153451.xml 2022-09-27T15:34:55.4464449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:55.4464845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:55.4473039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:55.4473389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:55.6019432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3384g8c 2022-09-27T15:34:55.6020300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3384g8c/_remote_module_non_scriptable.py 2022-09-27T15:34:55.8610262Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:34:55.8619642Z 2022-09-27T15:34:55.8620071Z Running tests... 2022-09-27T15:34:55.8620654Z ---------------------------------------------------------------------- 2022-09-27T15:34:56.1024312Z test_nested_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17082 2022-09-27T15:34:56.1051272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17083 2022-09-27T15:34:56.1078978Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17084 2022-09-27T15:34:56.1107505Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17085 2022-09-27T15:34:57.1663030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:57.1663669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:57.2007934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:57.2008586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:57.2137987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:57.2138600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:57.2256558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:57.2257150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:57.2285079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:57.2285709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:57.2508868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:57.2509448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:57.2633046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:57.2633691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:57.2652727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:57.2653363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:57.3285725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6nxre0p 2022-09-27T15:34:57.3286543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6nxre0p/_remote_module_non_scriptable.py 2022-09-27T15:34:57.3781411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1z8l_du 2022-09-27T15:34:57.3782234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1z8l_du/_remote_module_non_scriptable.py 2022-09-27T15:34:57.3904399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkao_av8l 2022-09-27T15:34:57.3905194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkao_av8l/_remote_module_non_scriptable.py 2022-09-27T15:34:57.3915640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6smslmli 2022-09-27T15:34:57.3917175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6smslmli/_remote_module_non_scriptable.py 2022-09-27T15:34:57.5847141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:34:57.6352538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:34:57.6448981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:34:57.6476607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:34:58.3163688Z ok (2.454s) 2022-09-27T15:34:58.3163922Z 2022-09-27T15:34:58.3164447Z ---------------------------------------------------------------------- 2022-09-27T15:34:58.3164890Z Ran 1 test in 2.454s 2022-09-27T15:34:58.3165006Z 2022-09-27T15:34:58.3165056Z OK 2022-09-27T15:34:58.3165148Z 2022-09-27T15:34:58.3165243Z Generating XML reports... 2022-09-27T15:34:58.3199583Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153455.xml 2022-09-27T15:34:59.7152413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:34:59.7152803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:34:59.7160818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:34:59.7161395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:34:59.8757738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6y26dcuw 2022-09-27T15:34:59.8758816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6y26dcuw/_remote_module_non_scriptable.py 2022-09-27T15:35:00.1365763Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:00.1375726Z 2022-09-27T15:35:00.1375875Z Running tests... 2022-09-27T15:35:00.1376227Z ---------------------------------------------------------------------- 2022-09-27T15:35:00.1397857Z test_no_grad_copy (__main__.TensorPipeDistAutogradTest) 2022-09-27T15:35:00.3809365Z Similar to test in test_autograd.py. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17305 2022-09-27T15:35:00.3836951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17306 2022-09-27T15:35:00.3864346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17307 2022-09-27T15:35:00.3893500Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17308 2022-09-27T15:35:01.4374734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:01.4375423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:01.4376232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:01.4376867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:01.4544792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:01.4545315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:01.4760517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:01.4761169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:01.4761909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:01.4762509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:01.4841116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:01.4841717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:01.4926873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:01.4927509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:01.5209068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:01.5209743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:01.6035481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjn9lf55f 2022-09-27T15:35:01.6036299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjn9lf55f/_remote_module_non_scriptable.py 2022-09-27T15:35:01.6036999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zdevsy0 2022-09-27T15:35:01.6040285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zdevsy0/_remote_module_non_scriptable.py 2022-09-27T15:35:01.6206431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8c7h69we 2022-09-27T15:35:01.6207228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8c7h69we/_remote_module_non_scriptable.py 2022-09-27T15:35:01.6486576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8gc21hw 2022-09-27T15:35:01.6487371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8gc21hw/_remote_module_non_scriptable.py 2022-09-27T15:35:01.8659428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:35:01.8667739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:35:01.8836764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:35:01.9130392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:35:02.4947132Z ok (2.357s) 2022-09-27T15:35:02.4947482Z 2022-09-27T15:35:02.4947845Z ---------------------------------------------------------------------- 2022-09-27T15:35:02.4948104Z Ran 1 test in 2.357s 2022-09-27T15:35:02.4948225Z 2022-09-27T15:35:02.4948286Z OK 2022-09-27T15:35:02.4948377Z 2022-09-27T15:35:02.4948475Z Generating XML reports... 2022-09-27T15:35:02.4983977Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153500.xml 2022-09-27T15:35:03.8822393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:03.8822811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:03.8830158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:03.8830567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:04.0413531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_9li5ty 2022-09-27T15:35:04.0414474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_9li5ty/_remote_module_non_scriptable.py 2022-09-27T15:35:04.3041595Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:04.3051043Z 2022-09-27T15:35:04.3051145Z Running tests... 2022-09-27T15:35:04.3051474Z ---------------------------------------------------------------------- 2022-09-27T15:35:04.5482034Z test_no_grad_copy_sparse (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17548 2022-09-27T15:35:04.5509607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17549 2022-09-27T15:35:04.5537047Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17550 2022-09-27T15:35:04.5565594Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17551 2022-09-27T15:35:05.7045806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:05.7046419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:05.7147807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:05.7148432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:05.7245805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:05.7246459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:05.7425843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:05.7426509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:05.7536304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:05.7536945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:05.7629807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:05.7630454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:05.7756816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:05.7757524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:05.8115724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:05.8116378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:05.8709015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2y_zm9u 2022-09-27T15:35:05.8709821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2y_zm9u/_remote_module_non_scriptable.py 2022-09-27T15:35:05.8819597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpixar3696 2022-09-27T15:35:05.8820318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpixar3696/_remote_module_non_scriptable.py 2022-09-27T15:35:05.8891573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpssyja10e 2022-09-27T15:35:05.8892240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpssyja10e/_remote_module_non_scriptable.py 2022-09-27T15:35:05.9389711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3_qevzw 2022-09-27T15:35:05.9390528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3_qevzw/_remote_module_non_scriptable.py 2022-09-27T15:35:06.1291324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:35:06.1411013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:35:06.1475260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:35:06.1953704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:35:06.4183255Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-09-27T15:35:06.4184366Z "Argument order of nn.functional.embedding_bag was changed. " 2022-09-27T15:35:06.4283670Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-09-27T15:35:06.4284755Z "Argument order of nn.functional.embedding_bag was changed. " 2022-09-27T15:35:06.4439956Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-09-27T15:35:06.4441042Z "Argument order of nn.functional.embedding_bag was changed. " 2022-09-27T15:35:06.4442487Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-09-27T15:35:06.4443500Z "Argument order of nn.functional.embedding_bag was changed. " 2022-09-27T15:35:06.8622236Z ok (2.557s) 2022-09-27T15:35:06.8623441Z 2022-09-27T15:35:06.8623967Z ---------------------------------------------------------------------- 2022-09-27T15:35:06.8624316Z Ran 1 test in 2.557s 2022-09-27T15:35:06.8624435Z 2022-09-27T15:35:06.8624499Z OK 2022-09-27T15:35:06.8624578Z 2022-09-27T15:35:06.8624672Z Generating XML reports... 2022-09-27T15:35:06.8661368Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153504.xml 2022-09-27T15:35:08.2243870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:08.2244527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:08.2252878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:08.2253344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:08.3801555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjiwbin0q 2022-09-27T15:35:08.3802479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjiwbin0q/_remote_module_non_scriptable.py 2022-09-27T15:35:08.6435755Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:08.6445993Z 2022-09-27T15:35:08.6446316Z Running tests... 2022-09-27T15:35:08.6446931Z ---------------------------------------------------------------------- 2022-09-27T15:35:08.8914319Z test_no_graph_with_tensors_not_require_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17803 2022-09-27T15:35:08.8942151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17804 2022-09-27T15:35:08.8969504Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17805 2022-09-27T15:35:08.8997588Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17806 2022-09-27T15:35:10.0038003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:10.0038656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:10.0067420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:10.0068112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:10.0201443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:10.0201951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:10.0282512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:10.0283048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:10.0419618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:10.0420384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:10.0450042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:10.0450677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:10.0579516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:10.0580211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:10.0648266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:10.0648919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:10.1693612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdvv80gwc 2022-09-27T15:35:10.1694881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdvv80gwc/_remote_module_non_scriptable.py 2022-09-27T15:35:10.1705752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvffr_5fb 2022-09-27T15:35:10.1707613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvffr_5fb/_remote_module_non_scriptable.py 2022-09-27T15:35:10.1847951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv70cwnnm 2022-09-27T15:35:10.1849444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv70cwnnm/_remote_module_non_scriptable.py 2022-09-27T15:35:10.1900680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5457v1h3 2022-09-27T15:35:10.1901942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5457v1h3/_remote_module_non_scriptable.py 2022-09-27T15:35:10.4253960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:35:10.4269890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:35:10.4398696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:35:10.4455989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:35:10.6948497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:35:10.7048082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:35:10.7150922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:35:10.7153045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:35:10.7154330Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:35:10.7155615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:35:10.7156754Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:35:10.7157859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:35:11.2055573Z ok (2.561s) 2022-09-27T15:35:11.2055775Z 2022-09-27T15:35:11.2056186Z ---------------------------------------------------------------------- 2022-09-27T15:35:11.2056533Z Ran 1 test in 2.561s 2022-09-27T15:35:11.2056666Z 2022-09-27T15:35:11.2056763Z OK 2022-09-27T15:35:11.2056886Z 2022-09-27T15:35:11.2057050Z Generating XML reports... 2022-09-27T15:35:11.2092419Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153508.xml 2022-09-27T15:35:12.5829573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:12.5829951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:12.5838539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:12.5839024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:12.7431242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qto7k97 2022-09-27T15:35:12.7432839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qto7k97/_remote_module_non_scriptable.py 2022-09-27T15:35:13.0027875Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:13.0037956Z 2022-09-27T15:35:13.0038059Z Running tests... 2022-09-27T15:35:13.0038784Z ---------------------------------------------------------------------- 2022-09-27T15:35:13.2440009Z test_no_graph_with_tensors_not_require_grad_remote (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18038 2022-09-27T15:35:13.2467186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18039 2022-09-27T15:35:13.2495064Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18040 2022-09-27T15:35:13.2523354Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18041 2022-09-27T15:35:14.3966443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:14.3967088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:14.4202822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:14.4203498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:14.4354055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:14.4354797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:14.4596779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:14.4597355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:14.4757771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:14.4758422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:14.4914199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:14.4915022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:14.5147235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:14.5147884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:14.5287584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:14.5288177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:14.5653405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bidz1q0 2022-09-27T15:35:14.5654161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bidz1q0/_remote_module_non_scriptable.py 2022-09-27T15:35:14.5872410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprvhp3lsr 2022-09-27T15:35:14.5873690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprvhp3lsr/_remote_module_non_scriptable.py 2022-09-27T15:35:14.6416471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5tywfk0 2022-09-27T15:35:14.6417284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5tywfk0/_remote_module_non_scriptable.py 2022-09-27T15:35:14.6564030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbzyjf_3 2022-09-27T15:35:14.6564976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbzyjf_3/_remote_module_non_scriptable.py 2022-09-27T15:35:14.8476284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:35:14.8503830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:35:14.9049006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:35:14.9157224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:35:15.1504811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:35:15.1606425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:35:15.1712040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:35:15.1712814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:35:15.1713253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:35:15.1714239Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:35:15.1715282Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:35:15.1716174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:35:15.7584524Z ok (2.754s) 2022-09-27T15:35:15.7584762Z 2022-09-27T15:35:15.7585244Z ---------------------------------------------------------------------- 2022-09-27T15:35:15.7585653Z Ran 1 test in 2.755s 2022-09-27T15:35:15.7585770Z 2022-09-27T15:35:15.7585833Z OK 2022-09-27T15:35:15.7585922Z 2022-09-27T15:35:15.7586015Z Generating XML reports... 2022-09-27T15:35:15.7621042Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153512.xml 2022-09-27T15:35:17.1349440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:17.1349866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:17.1358014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:17.1358381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:17.2938255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77xys6ar 2022-09-27T15:35:17.2939160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77xys6ar/_remote_module_non_scriptable.py 2022-09-27T15:35:17.5577987Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:17.5587195Z 2022-09-27T15:35:17.5587413Z Running tests... 2022-09-27T15:35:17.5588017Z ---------------------------------------------------------------------- 2022-09-27T15:35:17.8068144Z test_post_hooks (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18273 2022-09-27T15:35:17.8094744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18274 2022-09-27T15:35:17.8123296Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18275 2022-09-27T15:35:17.8158025Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18276 2022-09-27T15:35:18.9263437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:18.9264040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:18.9614436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:18.9615058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:18.9690357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:18.9691000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:18.9784408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:18.9785056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:18.9804855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:18.9805486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:19.0070203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:19.0070804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:19.0165242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:19.0166215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:19.0181206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:19.0181995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:19.0882851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqaa5ycs3 2022-09-27T15:35:19.0883625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqaa5ycs3/_remote_module_non_scriptable.py 2022-09-27T15:35:19.1336961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6waolnr9 2022-09-27T15:35:19.1338283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6waolnr9/_remote_module_non_scriptable.py 2022-09-27T15:35:19.1430754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphq6fml2d 2022-09-27T15:35:19.1432135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphq6fml2d/_remote_module_non_scriptable.py 2022-09-27T15:35:19.1436799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeu7cv3b5 2022-09-27T15:35:19.1438769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeu7cv3b5/_remote_module_non_scriptable.py 2022-09-27T15:35:19.3480628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:35:19.3953506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:35:19.4016435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:35:19.4031265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:35:20.0213844Z ok (2.462s) 2022-09-27T15:35:20.0214129Z 2022-09-27T15:35:20.0214626Z ---------------------------------------------------------------------- 2022-09-27T15:35:20.0214878Z Ran 1 test in 2.463s 2022-09-27T15:35:20.0214998Z 2022-09-27T15:35:20.0215046Z OK 2022-09-27T15:35:20.0215156Z 2022-09-27T15:35:20.0215252Z Generating XML reports... 2022-09-27T15:35:20.0252127Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153517.xml 2022-09-27T15:35:21.3639123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:21.3648273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:21.3649095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:21.3649715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:21.5256838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptw3czf6r 2022-09-27T15:35:21.5257888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptw3czf6r/_remote_module_non_scriptable.py 2022-09-27T15:35:21.7855998Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:21.7866099Z 2022-09-27T15:35:21.7866553Z Running tests... 2022-09-27T15:35:21.7867170Z ---------------------------------------------------------------------- 2022-09-27T15:35:22.0307738Z test_remote_complex_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18516 2022-09-27T15:35:22.0335700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18517 2022-09-27T15:35:22.0363430Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18518 2022-09-27T15:35:22.0391912Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18519 2022-09-27T15:35:23.0729811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:23.0730420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:23.0758981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:23.0759763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:23.0938357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:23.0938957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:23.1110553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:23.1111195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:23.1141082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:23.1141636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:23.1296920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:23.1297554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:23.1361915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:23.1362587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:23.1708242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:23.1708876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:23.2374335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeotjys8t 2022-09-27T15:35:23.2375113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeotjys8t/_remote_module_non_scriptable.py 2022-09-27T15:35:23.2407859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmt3fk_ri 2022-09-27T15:35:23.2409169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmt3fk_ri/_remote_module_non_scriptable.py 2022-09-27T15:35:23.2550803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xm5hugl 2022-09-27T15:35:23.2551888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xm5hugl/_remote_module_non_scriptable.py 2022-09-27T15:35:23.2959454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnaki4tbh 2022-09-27T15:35:23.2960622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnaki4tbh/_remote_module_non_scriptable.py 2022-09-27T15:35:23.4924483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:35:23.4990296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:35:23.5105184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:35:23.5518834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:35:24.2449289Z ok (2.458s) 2022-09-27T15:35:24.2449506Z 2022-09-27T15:35:24.2449895Z ---------------------------------------------------------------------- 2022-09-27T15:35:24.2450148Z Ran 1 test in 2.458s 2022-09-27T15:35:24.2450264Z 2022-09-27T15:35:24.2450325Z OK 2022-09-27T15:35:24.2450416Z 2022-09-27T15:35:24.2450497Z Generating XML reports... 2022-09-27T15:35:24.2486148Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153521.xml 2022-09-27T15:35:25.6126962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:25.6127601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:25.6136366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:25.6137012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:25.7696786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5r0cdyc 2022-09-27T15:35:25.7697450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5r0cdyc/_remote_module_non_scriptable.py 2022-09-27T15:35:26.0273398Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:26.0283445Z 2022-09-27T15:35:26.0283563Z Running tests... 2022-09-27T15:35:26.0284247Z ---------------------------------------------------------------------- 2022-09-27T15:35:26.2706522Z test_rpc_complex_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18739 2022-09-27T15:35:26.2733744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18740 2022-09-27T15:35:26.2760422Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18741 2022-09-27T15:35:26.2788468Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18742 2022-09-27T15:35:27.4309394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:27.4310105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:27.4667995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:27.4668682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:27.4681318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:27.4682010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:27.4742945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:27.4743591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:27.4902953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:27.4903489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:27.5055169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:27.5055764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:27.5136345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:27.5137012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:27.5263145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:27.5263825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:27.5969494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpacmd___w 2022-09-27T15:35:27.5970513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpacmd___w/_remote_module_non_scriptable.py 2022-09-27T15:35:27.6329560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7kvg50h 2022-09-27T15:35:27.6330377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7kvg50h/_remote_module_non_scriptable.py 2022-09-27T15:35:27.6401667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjk8k9rq 2022-09-27T15:35:27.6402955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjk8k9rq/_remote_module_non_scriptable.py 2022-09-27T15:35:27.6524272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1m_3pqe4 2022-09-27T15:35:27.6525884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1m_3pqe4/_remote_module_non_scriptable.py 2022-09-27T15:35:27.8546633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:35:27.8876963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:35:27.8945919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:35:27.9074042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:35:28.5846160Z ok (2.556s) 2022-09-27T15:35:28.5846403Z 2022-09-27T15:35:28.5846893Z ---------------------------------------------------------------------- 2022-09-27T15:35:28.5847294Z Ran 1 test in 2.556s 2022-09-27T15:35:28.5847412Z 2022-09-27T15:35:28.5847475Z OK 2022-09-27T15:35:28.5847567Z 2022-09-27T15:35:28.5847661Z Generating XML reports... 2022-09-27T15:35:28.5882301Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153526.xml 2022-09-27T15:35:29.9581898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:29.9582289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:29.9590052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:29.9590403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:30.1152828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxisjtzr2 2022-09-27T15:35:30.1153910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxisjtzr2/_remote_module_non_scriptable.py 2022-09-27T15:35:30.3755045Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:30.3764375Z 2022-09-27T15:35:30.3764516Z Running tests... 2022-09-27T15:35:30.3765183Z ---------------------------------------------------------------------- 2022-09-27T15:35:30.6201745Z test_thread_local_context_id (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18962 2022-09-27T15:35:30.6228648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18963 2022-09-27T15:35:30.6255375Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18964 2022-09-27T15:35:30.6284177Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18965 2022-09-27T15:35:31.7389742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:31.7390339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:31.7392396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:31.7392948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:31.7402399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:31.7403016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:31.7517888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:31.7518536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:31.7779777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:31.7780343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:31.7783883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:31.7784601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:31.7796696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:31.7797333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:31.7901182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:31.7901798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:31.9049894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurdzwp4f 2022-09-27T15:35:31.9050621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprgwsgxx4 2022-09-27T15:35:31.9051325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurdzwp4f/_remote_module_non_scriptable.py 2022-09-27T15:35:31.9052018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprgwsgxx4/_remote_module_non_scriptable.py 2022-09-27T15:35:31.9067011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3o20picq 2022-09-27T15:35:31.9068591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3o20picq/_remote_module_non_scriptable.py 2022-09-27T15:35:31.9171477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1cgstmqi 2022-09-27T15:35:31.9172475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1cgstmqi/_remote_module_non_scriptable.py 2022-09-27T15:35:32.1604593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:35:32.1621470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:35:32.1637127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:35:32.1754192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:35:33.8357657Z ok (3.459s) 2022-09-27T15:35:33.8357859Z 2022-09-27T15:35:33.8358368Z ---------------------------------------------------------------------- 2022-09-27T15:35:33.8358793Z Ran 1 test in 3.459s 2022-09-27T15:35:33.8358918Z 2022-09-27T15:35:33.8358965Z OK 2022-09-27T15:35:33.8359059Z 2022-09-27T15:35:33.8359156Z Generating XML reports... 2022-09-27T15:35:33.8395596Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153530.xml 2022-09-27T15:35:35.2131432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:35.2131831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:35.2139798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:35.2140160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:35.3743262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg34xmg0f 2022-09-27T15:35:35.3744016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg34xmg0f/_remote_module_non_scriptable.py 2022-09-27T15:35:35.6535782Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:35.6545679Z 2022-09-27T15:35:35.6546122Z Running tests... 2022-09-27T15:35:35.6546530Z ---------------------------------------------------------------------- 2022-09-27T15:35:35.9032546Z test_trainer_ps (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19205 2022-09-27T15:35:35.9059716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19206 2022-09-27T15:35:35.9088199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19207 2022-09-27T15:35:35.9117595Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19208 2022-09-27T15:35:37.0664808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:37.0665494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:37.0889333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:37.0890022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:37.1034194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:37.1175931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:37.1176873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:37.1177438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:37.1192413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:37.1193042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:37.1275790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:37.1276420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:37.1571430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:37.1590944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:37.1591925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:37.1592528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:37.2319426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2v1lls2t 2022-09-27T15:35:37.2320230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2v1lls2t/_remote_module_non_scriptable.py 2022-09-27T15:35:37.2562448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp75femcf 2022-09-27T15:35:37.2563635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp75femcf/_remote_module_non_scriptable.py 2022-09-27T15:35:37.2841594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeme5a7b7 2022-09-27T15:35:37.2842581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeme5a7b7/_remote_module_non_scriptable.py 2022-09-27T15:35:37.2871296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilynu6k6 2022-09-27T15:35:37.2873120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilynu6k6/_remote_module_non_scriptable.py 2022-09-27T15:35:37.4887884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:35:37.5165811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:35:37.5411452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:35:37.5627623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:35:38.3177825Z ok (2.663s) 2022-09-27T15:35:38.3178079Z 2022-09-27T15:35:38.3178591Z ---------------------------------------------------------------------- 2022-09-27T15:35:38.3178895Z Ran 1 test in 2.663s 2022-09-27T15:35:38.3179013Z 2022-09-27T15:35:38.3179075Z OK 2022-09-27T15:35:38.3179166Z 2022-09-27T15:35:38.3179260Z Generating XML reports... 2022-09-27T15:35:38.3216418Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153535.xml 2022-09-27T15:35:39.7042560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:39.7043142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:39.7050840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:39.7051303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:39.8646228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78sqjb7e 2022-09-27T15:35:39.8647487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78sqjb7e/_remote_module_non_scriptable.py 2022-09-27T15:35:40.1262613Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:40.1272764Z 2022-09-27T15:35:40.1272886Z Running tests... 2022-09-27T15:35:40.1273388Z ---------------------------------------------------------------------- 2022-09-27T15:35:40.3686109Z test_trainer_ps_torchscript_functions (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19448 2022-09-27T15:35:40.3713283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19449 2022-09-27T15:35:40.3740929Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19450 2022-09-27T15:35:40.3768965Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19451 2022-09-27T15:35:41.4852903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:41.4853532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:41.5193412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:41.5193999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:41.5229435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:41.5230040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:41.5240892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:41.5241577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:41.5459914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:41.5460436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:41.5582304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:41.5582965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:41.5629419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:41.5630070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:41.5831350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:41.5832108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:41.6530209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3skrzr4h 2022-09-27T15:35:41.6531025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3skrzr4h/_remote_module_non_scriptable.py 2022-09-27T15:35:41.6853513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20ysrcd7 2022-09-27T15:35:41.6854289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20ysrcd7/_remote_module_non_scriptable.py 2022-09-27T15:35:41.7085022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwzfu0de 2022-09-27T15:35:41.7086556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwzfu0de/_remote_module_non_scriptable.py 2022-09-27T15:35:41.7089846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppuss2v90 2022-09-27T15:35:41.7091962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppuss2v90/_remote_module_non_scriptable.py 2022-09-27T15:35:41.9128951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:35:41.9404009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:35:41.9678544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:35:41.9679136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:35:42.7830180Z ok (2.655s) 2022-09-27T15:35:42.7830503Z 2022-09-27T15:35:42.7830923Z ---------------------------------------------------------------------- 2022-09-27T15:35:42.7831175Z Ran 1 test in 2.656s 2022-09-27T15:35:42.7831297Z 2022-09-27T15:35:42.7831357Z OK 2022-09-27T15:35:42.7831441Z 2022-09-27T15:35:42.7831608Z Generating XML reports... 2022-09-27T15:35:42.7867163Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153540.xml 2022-09-27T15:35:44.1483759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:44.1484134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:44.1491767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:44.1492116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:44.3060603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpis2rv4kk 2022-09-27T15:35:44.3061672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpis2rv4kk/_remote_module_non_scriptable.py 2022-09-27T15:35:44.5679407Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:44.5688687Z 2022-09-27T15:35:44.5688797Z Running tests... 2022-09-27T15:35:44.5689240Z ---------------------------------------------------------------------- 2022-09-27T15:35:44.8191951Z test_worker_ids_recorded (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19691 2022-09-27T15:35:44.8219545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19692 2022-09-27T15:35:44.8246962Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19693 2022-09-27T15:35:44.8275478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19694 2022-09-27T15:35:45.8710754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:45.8711402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:45.8796165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:45.8796776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:45.8948430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:45.8949033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:45.9085010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:45.9085696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:45.9166068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:45.9166744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:45.9299309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:45.9299991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:45.9629293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:45.9629962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:45.9995678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:45.9996352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:46.0352765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt95xhied 2022-09-27T15:35:46.0354010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt95xhied/_remote_module_non_scriptable.py 2022-09-27T15:35:46.0420730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy67agrv4 2022-09-27T15:35:46.0422331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy67agrv4/_remote_module_non_scriptable.py 2022-09-27T15:35:46.0557519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmze5ruzt 2022-09-27T15:35:46.0558449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmze5ruzt/_remote_module_non_scriptable.py 2022-09-27T15:35:46.1242779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8cwo2nc 2022-09-27T15:35:46.1243549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8cwo2nc/_remote_module_non_scriptable.py 2022-09-27T15:35:46.2910303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:35:46.2960325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:35:46.3099190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:35:46.3799575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:35:47.1333964Z ok (2.564s) 2022-09-27T15:35:47.1334260Z 2022-09-27T15:35:47.1334764Z ---------------------------------------------------------------------- 2022-09-27T15:35:47.1335040Z Ran 1 test in 2.564s 2022-09-27T15:35:47.1335164Z 2022-09-27T15:35:47.1335225Z OK 2022-09-27T15:35:47.1335319Z 2022-09-27T15:35:47.1335423Z Generating XML reports... 2022-09-27T15:35:47.1371298Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153544.xml 2022-09-27T15:35:48.5133097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:48.5133708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:48.5142467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:48.5143092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:48.6733957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8enbcg9g 2022-09-27T15:35:48.6734830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8enbcg9g/_remote_module_non_scriptable.py 2022-09-27T15:35:48.9342468Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:48.9352416Z 2022-09-27T15:35:48.9352838Z Running tests... 2022-09-27T15:35:48.9353253Z ---------------------------------------------------------------------- 2022-09-27T15:35:49.1685396Z test_dist_optim (__main__.TensorPipeDistOptimizerTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/72997 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.233s) 2022-09-27T15:35:49.1686122Z 2022-09-27T15:35:49.1686351Z ---------------------------------------------------------------------- 2022-09-27T15:35:49.1686667Z Ran 1 test in 0.233s 2022-09-27T15:35:49.1686784Z 2022-09-27T15:35:49.1686856Z OK (skipped=1) 2022-09-27T15:35:49.1686963Z 2022-09-27T15:35:49.1687035Z Generating XML reports... 2022-09-27T15:35:49.1709419Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220927153548.xml 2022-09-27T15:35:50.4588786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:50.4589162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:50.4597230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:50.4597592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:50.6184359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsd885e6s 2022-09-27T15:35:50.6185082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsd885e6s/_remote_module_non_scriptable.py 2022-09-27T15:35:50.8798757Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:50.8808401Z 2022-09-27T15:35:50.8808503Z Running tests... 2022-09-27T15:35:50.8809549Z ---------------------------------------------------------------------- 2022-09-27T15:35:51.1245068Z test_dist_optim_exception (__main__.TensorPipeDistOptimizerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19924 2022-09-27T15:35:51.1271560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19925 2022-09-27T15:35:51.1299268Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19926 2022-09-27T15:35:51.1327217Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19927 2022-09-27T15:35:52.2814559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:52.2815240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:52.2890695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:52.2891330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:52.2966289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:52.2966867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:52.2986230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:52.2986736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:52.3222820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:52.3223476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:52.3289464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:52.3290126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:52.3358985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:52.3359652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:52.3381982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:52.3382576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:52.4509352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzxra_go 2022-09-27T15:35:52.4510227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzxra_go/_remote_module_non_scriptable.py 2022-09-27T15:35:52.4559414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2i4j3wxn 2022-09-27T15:35:52.4560984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2i4j3wxn/_remote_module_non_scriptable.py 2022-09-27T15:35:52.4636513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphzrz028o 2022-09-27T15:35:52.4637542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphzrz028o/_remote_module_non_scriptable.py 2022-09-27T15:35:52.4648921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7rkkjju 2022-09-27T15:35:52.4650848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7rkkjju/_remote_module_non_scriptable.py 2022-09-27T15:35:52.7116567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:35:52.7152148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:35:52.7229196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:35:52.7269455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:35:52.9268611Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-09-27T15:35:52.9365182Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-09-27T15:35:52.9368301Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:35:52.9368801Z ValueError('Error running optimizer.') 2022-09-27T15:35:52.9369209Z Traceback (most recent call last): 2022-09-27T15:35:52.9369959Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:52.9370555Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:52.9371370Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-09-27T15:35:52.9371918Z local_optim.step(autograd_ctx_id) 2022-09-27T15:35:52.9372617Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-09-27T15:35:52.9373142Z self.optim.step() 2022-09-27T15:35:52.9373587Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-09-27T15:35:52.9373860Z return func(*args, **kwargs) 2022-09-27T15:35:52.9374269Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-09-27T15:35:52.9374604Z raise ValueError("Error running optimizer.") 2022-09-27T15:35:52.9374828Z ValueError: Error running optimizer. 2022-09-27T15:35:52.9374964Z 2022-09-27T15:35:52.9389591Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:35:52.9390077Z ValueError('Error running optimizer.') 2022-09-27T15:35:52.9390457Z Traceback (most recent call last): 2022-09-27T15:35:52.9391203Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:52.9391888Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:52.9392858Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-09-27T15:35:52.9393223Z local_optim.step(autograd_ctx_id) 2022-09-27T15:35:52.9393840Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-09-27T15:35:52.9394436Z self.optim.step() 2022-09-27T15:35:52.9395336Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-09-27T15:35:52.9395923Z return func(*args, **kwargs) 2022-09-27T15:35:52.9396840Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-09-27T15:35:52.9397552Z raise ValueError("Error running optimizer.") 2022-09-27T15:35:52.9398041Z ValueError: Error running optimizer. 2022-09-27T15:35:52.9398333Z 2022-09-27T15:35:52.9477744Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-09-27T15:35:52.9479107Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-09-27T15:35:53.0017465Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:35:53.0018334Z ValueError('Error running optimizer.') 2022-09-27T15:35:53.0018785Z Traceback (most recent call last): 2022-09-27T15:35:53.0019666Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:53.0020367Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:53.0021326Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-09-27T15:35:53.0021945Z local_optim.step(autograd_ctx_id) 2022-09-27T15:35:53.0022752Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-09-27T15:35:53.0023331Z self.optim.step() 2022-09-27T15:35:53.0024058Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-09-27T15:35:53.0024628Z return func(*args, **kwargs) 2022-09-27T15:35:53.0025520Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-09-27T15:35:53.0026198Z raise ValueError("Error running optimizer.") 2022-09-27T15:35:53.0026693Z ValueError: Error running optimizer. 2022-09-27T15:35:53.0026979Z 2022-09-27T15:35:53.0027166Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:35:53.0027710Z ValueError('Error running optimizer.') 2022-09-27T15:35:53.0028153Z Traceback (most recent call last): 2022-09-27T15:35:53.0028963Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:53.0029649Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:53.0030568Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-09-27T15:35:53.0031213Z local_optim.step(autograd_ctx_id) 2022-09-27T15:35:53.0032099Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-09-27T15:35:53.0032689Z self.optim.step() 2022-09-27T15:35:53.0033416Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-09-27T15:35:53.0034321Z return func(*args, **kwargs) 2022-09-27T15:35:53.0035506Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-09-27T15:35:53.0036303Z raise ValueError("Error running optimizer.") 2022-09-27T15:35:53.0036795Z ValueError: Error running optimizer. 2022-09-27T15:35:53.0037083Z 2022-09-27T15:35:53.0348344Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:35:53.0357340Z ValueError('Error running optimizer.') 2022-09-27T15:35:53.0357731Z Traceback (most recent call last): 2022-09-27T15:35:53.0358460Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:53.0359049Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:53.0359854Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-09-27T15:35:53.0360422Z local_optim.step(autograd_ctx_id) 2022-09-27T15:35:53.0361183Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-09-27T15:35:53.0361757Z self.optim.step() 2022-09-27T15:35:53.0362637Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-09-27T15:35:53.0363216Z return func(*args, **kwargs) 2022-09-27T15:35:53.0364089Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-09-27T15:35:53.0364794Z raise ValueError("Error running optimizer.") 2022-09-27T15:35:53.0365285Z ValueError: Error running optimizer. 2022-09-27T15:35:53.0365570Z 2022-09-27T15:35:53.0365764Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:35:53.0366292Z ValueError('Error running optimizer.') 2022-09-27T15:35:53.0366759Z Traceback (most recent call last): 2022-09-27T15:35:53.0367593Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:53.0368269Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:53.0369208Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-09-27T15:35:53.0369857Z local_optim.step(autograd_ctx_id) 2022-09-27T15:35:53.0370645Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-09-27T15:35:53.0371043Z self.optim.step() 2022-09-27T15:35:53.0371807Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-09-27T15:35:53.0372374Z return func(*args, **kwargs) 2022-09-27T15:35:53.0373163Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-09-27T15:35:53.0373857Z raise ValueError("Error running optimizer.") 2022-09-27T15:35:53.0374333Z ValueError: Error running optimizer. 2022-09-27T15:35:53.0374608Z 2022-09-27T15:35:53.0374781Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:35:53.0375312Z ValueError('Error running optimizer.') 2022-09-27T15:35:53.0375743Z Traceback (most recent call last): 2022-09-27T15:35:53.0376402Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:53.0376901Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:53.0377647Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-09-27T15:35:53.0378197Z local_optim.step(autograd_ctx_id) 2022-09-27T15:35:53.0378858Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-09-27T15:35:53.0379350Z self.optim.step() 2022-09-27T15:35:53.0380042Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-09-27T15:35:53.0380813Z return func(*args, **kwargs) 2022-09-27T15:35:53.0381462Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-09-27T15:35:53.0382133Z raise ValueError("Error running optimizer.") 2022-09-27T15:35:53.0382496Z ValueError: Error running optimizer. 2022-09-27T15:35:53.0382716Z 2022-09-27T15:35:53.0382866Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:35:53.0383396Z ValueError('Error running optimizer.') 2022-09-27T15:35:53.0383856Z Traceback (most recent call last): 2022-09-27T15:35:53.0384675Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:53.0385369Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:53.0386312Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step 2022-09-27T15:35:53.0386966Z local_optim.step(autograd_ctx_id) 2022-09-27T15:35:53.0387674Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step 2022-09-27T15:35:53.0388222Z self.optim.step() 2022-09-27T15:35:53.0388958Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-09-27T15:35:53.0389504Z return func(*args, **kwargs) 2022-09-27T15:35:53.0390391Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-09-27T15:35:53.0391003Z raise ValueError("Error running optimizer.") 2022-09-27T15:35:53.0391599Z ValueError: Error running optimizer. 2022-09-27T15:35:53.0391832Z 2022-09-27T15:35:53.4400416Z ok (2.559s) 2022-09-27T15:35:53.4400681Z 2022-09-27T15:35:53.4401182Z ---------------------------------------------------------------------- 2022-09-27T15:35:53.4401460Z Ran 1 test in 2.559s 2022-09-27T15:35:53.4401579Z 2022-09-27T15:35:53.4401639Z OK 2022-09-27T15:35:53.4401735Z 2022-09-27T15:35:53.4401845Z Generating XML reports... 2022-09-27T15:35:53.4436795Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220927153550.xml 2022-09-27T15:35:54.8178699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:54.8179080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:54.8187635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:54.8187997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:54.9758188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmply70po84 2022-09-27T15:35:54.9759241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmply70po84/_remote_module_non_scriptable.py 2022-09-27T15:35:55.2365037Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:55.2375118Z 2022-09-27T15:35:55.2375261Z Running tests... 2022-09-27T15:35:55.2375672Z ---------------------------------------------------------------------- 2022-09-27T15:35:55.4783792Z test_dist_optim_exception_on_constructor (__main__.TensorPipeDistOptimizerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20167 2022-09-27T15:35:55.4810954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20168 2022-09-27T15:35:55.4838511Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20169 2022-09-27T15:35:55.4867552Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20170 2022-09-27T15:35:56.6142685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:56.6143315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:56.6148918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:56.6149540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:56.6181169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:56.6181757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:56.6249468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:56.6249970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:56.6543857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:56.6549618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:56.6550617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:56.6551321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:56.6586598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:56.6587335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:56.6647531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:56.6648181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:56.7826072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpamv7eo_j 2022-09-27T15:35:56.7826698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qz6tro9 2022-09-27T15:35:56.7827308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpamv7eo_j/_remote_module_non_scriptable.py 2022-09-27T15:35:56.7827979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qz6tro9/_remote_module_non_scriptable.py 2022-09-27T15:35:56.7867128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadkvkkgc 2022-09-27T15:35:56.7868586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadkvkkgc/_remote_module_non_scriptable.py 2022-09-27T15:35:56.7917254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbrfk9dlb 2022-09-27T15:35:56.7918819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbrfk9dlb/_remote_module_non_scriptable.py 2022-09-27T15:35:57.0355301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:35:57.0389427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:35:57.0433604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:35:57.0472770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:35:57.2672419Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-09-27T15:35:57.2683823Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:35:57.2684659Z ValueError('Error creating optimizer.') 2022-09-27T15:35:57.2685283Z Traceback (most recent call last): 2022-09-27T15:35:57.2686326Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:57.2687209Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:57.2688827Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-09-27T15:35:57.2689867Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-09-27T15:35:57.2691068Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-09-27T15:35:57.2691786Z **kwargs) 2022-09-27T15:35:57.2692925Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-09-27T15:35:57.2693797Z raise ValueError("Error creating optimizer.") 2022-09-27T15:35:57.2694450Z ValueError: Error creating optimizer. 2022-09-27T15:35:57.2694736Z 2022-09-27T15:35:57.2694908Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:35:57.2695616Z ValueError('Error creating optimizer.') 2022-09-27T15:35:57.2696241Z Traceback (most recent call last): 2022-09-27T15:35:57.2697113Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:57.2697884Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:57.2698789Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-09-27T15:35:57.2699565Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-09-27T15:35:57.2700574Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-09-27T15:35:57.2701127Z **kwargs) 2022-09-27T15:35:57.2701969Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-09-27T15:35:57.2702675Z raise ValueError("Error creating optimizer.") 2022-09-27T15:35:57.2703163Z ValueError: Error creating optimizer. 2022-09-27T15:35:57.2703448Z 2022-09-27T15:35:57.2765154Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-09-27T15:35:57.2867198Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-09-27T15:35:57.2870272Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-09-27T15:35:57.2936060Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:35:57.2937874Z ValueError('Error creating optimizer.') 2022-09-27T15:35:57.2938350Z Traceback (most recent call last): 2022-09-27T15:35:57.2939195Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:57.2939909Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:57.2940867Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-09-27T15:35:57.2941482Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-09-27T15:35:57.2942379Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-09-27T15:35:57.2943268Z **kwargs) 2022-09-27T15:35:57.2944123Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-09-27T15:35:57.2944914Z raise ValueError("Error creating optimizer.") 2022-09-27T15:35:57.2945418Z ValueError: Error creating optimizer. 2022-09-27T15:35:57.2947258Z 2022-09-27T15:35:57.2948575Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:35:57.2949137Z ValueError('Error creating optimizer.') 2022-09-27T15:35:57.2949597Z Traceback (most recent call last): 2022-09-27T15:35:57.2950424Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:57.2951143Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:57.2952151Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-09-27T15:35:57.2952877Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-09-27T15:35:57.2953767Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-09-27T15:35:57.2954324Z **kwargs) 2022-09-27T15:35:57.2955320Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-09-27T15:35:57.2956026Z raise ValueError("Error creating optimizer.") 2022-09-27T15:35:57.2956531Z ValueError: Error creating optimizer. 2022-09-27T15:35:57.2956813Z 2022-09-27T15:35:57.2957315Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:35:57.2957866Z ValueError('Error creating optimizer.') 2022-09-27T15:35:57.2958317Z Traceback (most recent call last): 2022-09-27T15:35:57.2958753Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:57.2959098Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:57.2959548Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-09-27T15:35:57.2959899Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-09-27T15:35:57.2960314Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-09-27T15:35:57.2960580Z **kwargs) 2022-09-27T15:35:57.2960979Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-09-27T15:35:57.2961318Z raise ValueError("Error creating optimizer.") 2022-09-27T15:35:57.2961550Z ValueError: Error creating optimizer. 2022-09-27T15:35:57.2961684Z 2022-09-27T15:35:57.3310504Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:35:57.3311104Z ValueError('Error creating optimizer.') 2022-09-27T15:35:57.3311554Z Traceback (most recent call last): 2022-09-27T15:35:57.3312318Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:57.3312886Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:57.3313674Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-09-27T15:35:57.3314265Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-09-27T15:35:57.3315117Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-09-27T15:35:57.3315566Z **kwargs) 2022-09-27T15:35:57.3316224Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-09-27T15:35:57.3316804Z raise ValueError("Error creating optimizer.") 2022-09-27T15:35:57.3317218Z ValueError: Error creating optimizer. 2022-09-27T15:35:57.3317450Z 2022-09-27T15:35:57.3340898Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:35:57.3342974Z ValueError('Error creating optimizer.') 2022-09-27T15:35:57.3343661Z Traceback (most recent call last): 2022-09-27T15:35:57.3344303Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:57.3344774Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:57.3345366Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-09-27T15:35:57.3345765Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-09-27T15:35:57.3346189Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-09-27T15:35:57.3346501Z **kwargs) 2022-09-27T15:35:57.3346904Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-09-27T15:35:57.3347304Z raise ValueError("Error creating optimizer.") 2022-09-27T15:35:57.3347533Z ValueError: Error creating optimizer. 2022-09-27T15:35:57.3347674Z 2022-09-27T15:35:57.3351361Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:35:57.3352035Z ValueError('Error creating optimizer.') 2022-09-27T15:35:57.3352423Z Traceback (most recent call last): 2022-09-27T15:35:57.3353127Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:35:57.3353469Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:35:57.3353911Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer 2022-09-27T15:35:57.3354363Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-09-27T15:35:57.3355173Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__ 2022-09-27T15:35:57.3355560Z **kwargs) 2022-09-27T15:35:57.3356204Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-09-27T15:35:57.3356731Z raise ValueError("Error creating optimizer.") 2022-09-27T15:35:57.3357076Z ValueError: Error creating optimizer. 2022-09-27T15:35:57.3357277Z 2022-09-27T15:35:57.7925264Z ok (2.555s) 2022-09-27T15:35:57.7925497Z 2022-09-27T15:35:57.7926017Z ---------------------------------------------------------------------- 2022-09-27T15:35:57.7926381Z Ran 1 test in 2.555s 2022-09-27T15:35:57.7926501Z 2022-09-27T15:35:57.7926562Z OK 2022-09-27T15:35:57.7926655Z 2022-09-27T15:35:57.7926752Z Generating XML reports... 2022-09-27T15:35:57.7961884Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220927153555.xml 2022-09-27T15:35:59.1507091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:35:59.1507481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:35:59.1516566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:35:59.1516926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:35:59.3109362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5es1h_mc 2022-09-27T15:35:59.3110587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5es1h_mc/_remote_module_non_scriptable.py 2022-09-27T15:35:59.5726179Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:35:59.5735797Z 2022-09-27T15:35:59.5735897Z Running tests... 2022-09-27T15:35:59.5736670Z ---------------------------------------------------------------------- 2022-09-27T15:35:59.8079219Z test_dist_optim_none_grads (__main__.TensorPipeDistOptimizerTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77075 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.234s) 2022-09-27T15:35:59.8080247Z 2022-09-27T15:35:59.8080605Z ---------------------------------------------------------------------- 2022-09-27T15:35:59.8081032Z Ran 1 test in 0.234s 2022-09-27T15:35:59.8081234Z 2022-09-27T15:35:59.8081357Z OK (skipped=1) 2022-09-27T15:35:59.8081523Z 2022-09-27T15:35:59.8081667Z Generating XML reports... 2022-09-27T15:35:59.8104099Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220927153559.xml 2022-09-27T15:36:01.1149746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:01.1150154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:01.1158158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:01.1158553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:01.2748949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxwvtqvs 2022-09-27T15:36:01.2749848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxwvtqvs/_remote_module_non_scriptable.py 2022-09-27T15:36:01.5358613Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:01.5368013Z 2022-09-27T15:36:01.5368120Z Running tests... 2022-09-27T15:36:01.5369101Z ---------------------------------------------------------------------- 2022-09-27T15:36:01.7861171Z test_dist_backward (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20400 2022-09-27T15:36:01.7888462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20401 2022-09-27T15:36:01.7916663Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20402 2022-09-27T15:36:01.7945226Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20403 2022-09-27T15:36:02.9137080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:02.9137703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:02.9323654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:02.9324535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:02.9325830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:02.9326576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:02.9434081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:02.9434850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:02.9528324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:02.9528897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:02.9732790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:02.9733737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:02.9735057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:02.9735701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:02.9841108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:02.9842016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:03.0814582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeidu5rah 2022-09-27T15:36:03.0815597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeidu5rah/_remote_module_non_scriptable.py 2022-09-27T15:36:03.1002609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8mq_lsf 2022-09-27T15:36:03.1003330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplfjm2m9w 2022-09-27T15:36:03.1004104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8mq_lsf/_remote_module_non_scriptable.py 2022-09-27T15:36:03.1005222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplfjm2m9w/_remote_module_non_scriptable.py 2022-09-27T15:36:03.1121739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76hkje8l 2022-09-27T15:36:03.1122618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76hkje8l/_remote_module_non_scriptable.py 2022-09-27T15:36:03.3386687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:36:03.3544204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:03.3697606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:36:03.3731560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:36:04.1002919Z ok (2.563s) 2022-09-27T15:36:04.1003159Z 2022-09-27T15:36:04.1003560Z ---------------------------------------------------------------------- 2022-09-27T15:36:04.1004011Z Ran 1 test in 2.563s 2022-09-27T15:36:04.1004228Z 2022-09-27T15:36:04.1004343Z OK 2022-09-27T15:36:04.1004454Z 2022-09-27T15:36:04.1004550Z Generating XML reports... 2022-09-27T15:36:04.1039798Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220927153601.xml 2022-09-27T15:36:05.4740485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:05.4740865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:05.4747869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:05.4748376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:05.6315931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__r_pw_9 2022-09-27T15:36:05.6316841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__r_pw_9/_remote_module_non_scriptable.py 2022-09-27T15:36:05.8925408Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:05.8935280Z 2022-09-27T15:36:05.8935410Z Running tests... 2022-09-27T15:36:05.8935844Z ---------------------------------------------------------------------- 2022-09-27T15:36:06.1340275Z test_get_gradients (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20633 2022-09-27T15:36:06.1367418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20634 2022-09-27T15:36:06.1394722Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20635 2022-09-27T15:36:06.1422673Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20636 2022-09-27T15:36:07.3000543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:07.3001159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:07.3048964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:07.3049640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:07.3087474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:07.3088226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:07.3147256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:07.3147764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:07.3382575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:07.3383321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:07.3437555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:07.3438191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:07.3474810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:07.3475492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:07.3526758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:07.3527413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:07.4655218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0c5d4sq1 2022-09-27T15:36:07.4655954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0c5d4sq1/_remote_module_non_scriptable.py 2022-09-27T15:36:07.4708175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qpdhus2 2022-09-27T15:36:07.4709517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qpdhus2/_remote_module_non_scriptable.py 2022-09-27T15:36:07.4747026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lw5qqqq 2022-09-27T15:36:07.4748563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lw5qqqq/_remote_module_non_scriptable.py 2022-09-27T15:36:07.4970083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dklrkmf 2022-09-27T15:36:07.4970957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dklrkmf/_remote_module_non_scriptable.py 2022-09-27T15:36:07.7210910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:36:07.7251178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:07.7337214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:36:07.7553055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:36:08.4481431Z ok (2.554s) 2022-09-27T15:36:08.4481689Z 2022-09-27T15:36:08.4482209Z ---------------------------------------------------------------------- 2022-09-27T15:36:08.4482644Z Ran 1 test in 2.555s 2022-09-27T15:36:08.4482773Z 2022-09-27T15:36:08.4482833Z OK 2022-09-27T15:36:08.4482926Z 2022-09-27T15:36:08.4483025Z Generating XML reports... 2022-09-27T15:36:08.4518889Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220927153605.xml 2022-09-27T15:36:09.8317009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:09.8317643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:09.8325891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:09.8326525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:09.9909493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiqpc1axk 2022-09-27T15:36:09.9910199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiqpc1axk/_remote_module_non_scriptable.py 2022-09-27T15:36:10.2514183Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:10.2524471Z 2022-09-27T15:36:10.2524891Z Running tests... 2022-09-27T15:36:10.2525308Z ---------------------------------------------------------------------- 2022-09-27T15:36:10.4977260Z test_jit_fork_within_context (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20876 2022-09-27T15:36:10.5004622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20877 2022-09-27T15:36:10.5031377Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20878 2022-09-27T15:36:10.5060075Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20879 2022-09-27T15:36:11.6706794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:11.6707457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:11.7063453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:11.7064042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:11.7105942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:11.7106538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:11.7161395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:11.7162014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:11.7386564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:11.7387262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:11.7476450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:11.7476998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:11.7535194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:11.7535752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:11.7747980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:11.7748620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:11.8341334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7m1ool12 2022-09-27T15:36:11.8342163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7m1ool12/_remote_module_non_scriptable.py 2022-09-27T15:36:11.8753150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb9g0fupn 2022-09-27T15:36:11.8753915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb9g0fupn/_remote_module_non_scriptable.py 2022-09-27T15:36:11.8792391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8f1iv50 2022-09-27T15:36:11.8794423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8f1iv50/_remote_module_non_scriptable.py 2022-09-27T15:36:11.9015170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0w_qllmg 2022-09-27T15:36:11.9016788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0w_qllmg/_remote_module_non_scriptable.py 2022-09-27T15:36:12.0923271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:36:12.1341126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:36:12.1508761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:36:12.1713745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:12.9120156Z ok (2.659s) 2022-09-27T15:36:12.9120402Z 2022-09-27T15:36:12.9120726Z ---------------------------------------------------------------------- 2022-09-27T15:36:12.9120977Z Ran 1 test in 2.659s 2022-09-27T15:36:12.9121095Z 2022-09-27T15:36:12.9121143Z OK 2022-09-27T15:36:12.9121234Z 2022-09-27T15:36:12.9121327Z Generating XML reports... 2022-09-27T15:36:12.9156195Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220927153610.xml 2022-09-27T15:36:14.2776871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:14.2777270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:14.2785077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:14.2785461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:14.4331532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1e8z1swb 2022-09-27T15:36:14.4332508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1e8z1swb/_remote_module_non_scriptable.py 2022-09-27T15:36:14.7042879Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:14.7052267Z 2022-09-27T15:36:14.7052499Z Running tests... 2022-09-27T15:36:14.7052888Z ---------------------------------------------------------------------- 2022-09-27T15:36:14.9459380Z test_restore_context_after_swtich_to_jit_thread (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21119 2022-09-27T15:36:14.9487053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21120 2022-09-27T15:36:14.9513755Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21121 2022-09-27T15:36:14.9541798Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21122 2022-09-27T15:36:16.0603400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:16.0604085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:16.0619794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:16.0620450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:16.0647215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:16.0647839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:16.0954346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:16.0955355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:16.0995701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:16.0996313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:16.1009954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:16.1011204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:16.1035737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:16.1036608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:16.1325235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:16.1326247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:16.2269371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwle749a 2022-09-27T15:36:16.2270345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwle749a/_remote_module_non_scriptable.py 2022-09-27T15:36:16.2275804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7pcyhgd 2022-09-27T15:36:16.2277470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7pcyhgd/_remote_module_non_scriptable.py 2022-09-27T15:36:16.2317951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3fllzjb 2022-09-27T15:36:16.2320715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3fllzjb/_remote_module_non_scriptable.py 2022-09-27T15:36:16.2580456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc52g0x79 2022-09-27T15:36:16.2581684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc52g0x79/_remote_module_non_scriptable.py 2022-09-27T15:36:16.4852022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:36:16.5126461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:16.5594791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:36:16.5626073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:36:17.2599671Z ok (2.554s) 2022-09-27T15:36:17.2599854Z 2022-09-27T15:36:17.2600162Z ---------------------------------------------------------------------- 2022-09-27T15:36:17.2600461Z Ran 1 test in 2.555s 2022-09-27T15:36:17.2600579Z 2022-09-27T15:36:17.2600627Z OK 2022-09-27T15:36:17.2600728Z 2022-09-27T15:36:17.2600825Z Generating XML reports... 2022-09-27T15:36:17.2635550Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220927153614.xml 2022-09-27T15:36:18.6230025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:18.6230412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:18.6238480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:18.6238827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:18.7790586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr54xrsew 2022-09-27T15:36:18.7792180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr54xrsew/_remote_module_non_scriptable.py 2022-09-27T15:36:19.0535420Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:19.0545160Z 2022-09-27T15:36:19.0545262Z Running tests... 2022-09-27T15:36:19.0546291Z ---------------------------------------------------------------------- 2022-09-27T15:36:19.2958442Z test_add_done_callback (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21352 2022-09-27T15:36:19.2985818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21353 2022-09-27T15:36:19.3013187Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21354 2022-09-27T15:36:19.3041936Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21355 2022-09-27T15:36:20.4234889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:20.4235534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:20.4307914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:20.4308727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:20.4587997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:20.4588658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:20.4663546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:20.4664181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:20.4974317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:20.4974940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:20.5229182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:20.5229835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:20.5327860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:20.5328511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:20.5587027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:20.5587694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:20.5856011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp182_f4p8 2022-09-27T15:36:20.5856698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp182_f4p8/_remote_module_non_scriptable.py 2022-09-27T15:36:20.5943289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy67ljr7v 2022-09-27T15:36:20.5944886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy67ljr7v/_remote_module_non_scriptable.py 2022-09-27T15:36:20.6600620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpflzm66ux 2022-09-27T15:36:20.6601374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpflzm66ux/_remote_module_non_scriptable.py 2022-09-27T15:36:20.6947338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpep9zpwmk 2022-09-27T15:36:20.6948172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpep9zpwmk/_remote_module_non_scriptable.py 2022-09-27T15:36:20.8433174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:20.8622637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:36:20.9208376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:36:20.9509650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:36:21.7101420Z ok (2.655s) 2022-09-27T15:36:21.7101744Z 2022-09-27T15:36:21.7102263Z ---------------------------------------------------------------------- 2022-09-27T15:36:21.7102634Z Ran 1 test in 2.656s 2022-09-27T15:36:21.7102750Z 2022-09-27T15:36:21.7102811Z OK 2022-09-27T15:36:21.7102903Z 2022-09-27T15:36:21.7102984Z Generating XML reports... 2022-09-27T15:36:21.7138038Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153619.xml 2022-09-27T15:36:23.0912063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:23.0912434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:23.0920375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:23.0921007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:23.2459636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_r03yn3 2022-09-27T15:36:23.2460735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_r03yn3/_remote_module_non_scriptable.py 2022-09-27T15:36:23.5071643Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:23.5081761Z 2022-09-27T15:36:23.5081860Z Running tests... 2022-09-27T15:36:23.5083004Z ---------------------------------------------------------------------- 2022-09-27T15:36:23.7558476Z test_all_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21591 2022-09-27T15:36:23.7585735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21592 2022-09-27T15:36:23.7613885Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21593 2022-09-27T15:36:23.7642901Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21594 2022-09-27T15:36:24.8917689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:24.8918329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:24.9025287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:24.9025878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:24.9251643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:24.9252371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:24.9291986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:24.9292710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:24.9406195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:24.9406757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:24.9628270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:24.9628987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:24.9686399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:24.9687026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:25.0055194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:25.0055904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:25.0588017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_d50tqcx 2022-09-27T15:36:25.0590444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_d50tqcx/_remote_module_non_scriptable.py 2022-09-27T15:36:25.0681115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3cqhf8yw 2022-09-27T15:36:25.0682377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3cqhf8yw/_remote_module_non_scriptable.py 2022-09-27T15:36:25.1075521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpob7_gyy5 2022-09-27T15:36:25.1076411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpob7_gyy5/_remote_module_non_scriptable.py 2022-09-27T15:36:25.1347945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmoxbrtqd 2022-09-27T15:36:25.1348958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmoxbrtqd/_remote_module_non_scriptable.py 2022-09-27T15:36:25.3161732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:36:25.3249333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:36:25.3661917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:25.3937515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:36:26.0701639Z ok (2.562s) 2022-09-27T15:36:26.0701948Z 2022-09-27T15:36:26.0702470Z ---------------------------------------------------------------------- 2022-09-27T15:36:26.0702762Z Ran 1 test in 2.562s 2022-09-27T15:36:26.0702883Z 2022-09-27T15:36:26.0702949Z OK 2022-09-27T15:36:26.0703029Z 2022-09-27T15:36:26.0703127Z Generating XML reports... 2022-09-27T15:36:26.0737956Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153623.xml 2022-09-27T15:36:27.4517776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:27.4518171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:27.4525724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:27.4526093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:27.6125870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_r_r44vo 2022-09-27T15:36:27.6126616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_r_r44vo/_remote_module_non_scriptable.py 2022-09-27T15:36:27.8744445Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:27.8753810Z 2022-09-27T15:36:27.8753941Z Running tests... 2022-09-27T15:36:27.8754426Z ---------------------------------------------------------------------- 2022-09-27T15:36:28.1251381Z test_args_and_kwargs_contain_different_types (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21818 2022-09-27T15:36:28.1278762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21819 2022-09-27T15:36:28.1321891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21820 2022-09-27T15:36:28.1371814Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21821 2022-09-27T15:36:29.2239876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:29.2240520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:29.2448677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:29.2449316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:29.2454963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:29.2455576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:29.2597958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:29.2598622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:29.2624141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:29.2624715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:29.2844407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:29.2844997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:29.2846043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:29.2846667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:29.2971186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:29.2971813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:29.3883992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwh56jmya 2022-09-27T15:36:29.3884775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwh56jmya/_remote_module_non_scriptable.py 2022-09-27T15:36:29.4099341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4a_wf8zw 2022-09-27T15:36:29.4101142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4a_wf8zw/_remote_module_non_scriptable.py 2022-09-27T15:36:29.4101873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65gi7whw 2022-09-27T15:36:29.4103139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65gi7whw/_remote_module_non_scriptable.py 2022-09-27T15:36:29.4222122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwaxyeij9 2022-09-27T15:36:29.4222830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwaxyeij9/_remote_module_non_scriptable.py 2022-09-27T15:36:29.6516364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:36:29.6697061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:36:29.6702307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:36:29.6804387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:30.4429972Z ok (2.567s) 2022-09-27T15:36:30.4430183Z 2022-09-27T15:36:30.4430495Z ---------------------------------------------------------------------- 2022-09-27T15:36:30.4430745Z Ran 1 test in 2.568s 2022-09-27T15:36:30.4430879Z 2022-09-27T15:36:30.4430941Z OK 2022-09-27T15:36:30.4431034Z 2022-09-27T15:36:30.4431113Z Generating XML reports... 2022-09-27T15:36:30.4465724Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153627.xml 2022-09-27T15:36:31.8140277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:31.8140661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:31.8148462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:31.8148830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:31.9744980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdlt0bi2 2022-09-27T15:36:31.9746298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdlt0bi2/_remote_module_non_scriptable.py 2022-09-27T15:36:32.2369347Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:32.2378604Z 2022-09-27T15:36:32.2378911Z Running tests... 2022-09-27T15:36:32.2379538Z ---------------------------------------------------------------------- 2022-09-27T15:36:32.4865765Z test_args_kwargs_are_neither_passed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22045 2022-09-27T15:36:32.4892663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22046 2022-09-27T15:36:32.4920616Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22047 2022-09-27T15:36:32.4949352Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22048 2022-09-27T15:36:33.5567190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:33.5567873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:33.5722124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:33.5722766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:33.5947680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:33.5948334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:33.6025189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:33.6025798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:33.6101795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:33.6102432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:33.6138580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:33.6139228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:33.6407090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:33.6407728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:33.6513932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:33.6514708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:33.7214819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyag5ptv 2022-09-27T15:36:33.7215620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyag5ptv/_remote_module_non_scriptable.py 2022-09-27T15:36:33.7371888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gzk0hy2 2022-09-27T15:36:33.7372661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gzk0hy2/_remote_module_non_scriptable.py 2022-09-27T15:36:33.7681045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzs_vq071 2022-09-27T15:36:33.7681830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzs_vq071/_remote_module_non_scriptable.py 2022-09-27T15:36:33.7783148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuruhztde 2022-09-27T15:36:33.7784530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuruhztde/_remote_module_non_scriptable.py 2022-09-27T15:36:33.9786528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:36:33.9929245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:34.0275970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:36:34.0370512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:36:34.7007440Z ok (2.462s) 2022-09-27T15:36:34.7007693Z 2022-09-27T15:36:34.7008035Z ---------------------------------------------------------------------- 2022-09-27T15:36:34.7008306Z Ran 1 test in 2.463s 2022-09-27T15:36:34.7008426Z 2022-09-27T15:36:34.7008488Z OK 2022-09-27T15:36:34.7008584Z 2022-09-27T15:36:34.7008665Z Generating XML reports... 2022-09-27T15:36:34.7043170Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153632.xml 2022-09-27T15:36:36.0902589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:36.0903201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:36.0910914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:36.0911622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:36.2473308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfte1bnie 2022-09-27T15:36:36.2474777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfte1bnie/_remote_module_non_scriptable.py 2022-09-27T15:36:36.5053566Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:36.5062618Z 2022-09-27T15:36:36.5062763Z Running tests... 2022-09-27T15:36:36.5063768Z ---------------------------------------------------------------------- 2022-09-27T15:36:36.7468557Z test_async_function_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22272 2022-09-27T15:36:36.7495041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22273 2022-09-27T15:36:36.7521726Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22274 2022-09-27T15:36:36.7551521Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22275 2022-09-27T15:36:37.8446228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:37.8446857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:37.8627995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:37.8628669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:37.8809807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:37.8810475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:37.8846840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:37.8847525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:37.8848354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:37.8848964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:37.9002499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:37.9003045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:37.9220759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:37.9221388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:37.9223524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:37.9224159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:38.0078767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz64_ck3w 2022-09-27T15:36:38.0079624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz64_ck3w/_remote_module_non_scriptable.py 2022-09-27T15:36:38.0269434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1hioxre 2022-09-27T15:36:38.0270240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1hioxre/_remote_module_non_scriptable.py 2022-09-27T15:36:38.0499242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzm1ot9v8 2022-09-27T15:36:38.0500055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzm1ot9v8/_remote_module_non_scriptable.py 2022-09-27T15:36:38.0500942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83wcgmm_ 2022-09-27T15:36:38.0502913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83wcgmm_/_remote_module_non_scriptable.py 2022-09-27T15:36:38.2632828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:36:38.2830597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:38.3059354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:36:38.3076858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:36:38.9608683Z ok (2.454s) 2022-09-27T15:36:38.9608961Z 2022-09-27T15:36:38.9609400Z ---------------------------------------------------------------------- 2022-09-27T15:36:38.9609784Z Ran 1 test in 2.455s 2022-09-27T15:36:38.9609889Z 2022-09-27T15:36:38.9609984Z OK 2022-09-27T15:36:38.9610119Z 2022-09-27T15:36:38.9610266Z Generating XML reports... 2022-09-27T15:36:38.9648696Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153636.xml 2022-09-27T15:36:40.3326469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:40.3326857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:40.3335674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:40.3336160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:40.4933755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxceuu6to 2022-09-27T15:36:40.4934587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxceuu6to/_remote_module_non_scriptable.py 2022-09-27T15:36:40.7555313Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:40.7565110Z 2022-09-27T15:36:40.7565204Z Running tests... 2022-09-27T15:36:40.7565602Z ---------------------------------------------------------------------- 2022-09-27T15:36:41.0034087Z test_async_function_remote_multi (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22495 2022-09-27T15:36:41.0062051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22496 2022-09-27T15:36:41.0090303Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22497 2022-09-27T15:36:41.0118865Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22498 2022-09-27T15:36:42.0936690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:42.0937355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:42.1212197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:42.1212838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:42.1304822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:42.1305428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:42.1355317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:42.1355931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:42.1587231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:42.1587870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:42.1713579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:42.1714260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:42.1783805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:42.1784400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:42.2123170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:42.2123837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:42.2581008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfyk9sq53 2022-09-27T15:36:42.2581802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfyk9sq53/_remote_module_non_scriptable.py 2022-09-27T15:36:42.2852724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_f4hq523 2022-09-27T15:36:42.2853578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_f4hq523/_remote_module_non_scriptable.py 2022-09-27T15:36:42.2975162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3p3ed67 2022-09-27T15:36:42.2975904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3p3ed67/_remote_module_non_scriptable.py 2022-09-27T15:36:42.3379450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjn9b7g1m 2022-09-27T15:36:42.3380298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjn9b7g1m/_remote_module_non_scriptable.py 2022-09-27T15:36:42.5134165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:36:42.5404505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:42.5515503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:36:42.5959394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:36:43.3178121Z ok (2.561s) 2022-09-27T15:36:43.3178365Z 2022-09-27T15:36:43.3178867Z ---------------------------------------------------------------------- 2022-09-27T15:36:43.3179249Z Ran 1 test in 2.561s 2022-09-27T15:36:43.3179367Z 2022-09-27T15:36:43.3179425Z OK 2022-09-27T15:36:43.3179517Z 2022-09-27T15:36:43.3179608Z Generating XML reports... 2022-09-27T15:36:43.3213913Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153640.xml 2022-09-27T15:36:44.6916822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:44.6917205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:44.6925629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:44.6925985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:44.8486126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptl3cas5c 2022-09-27T15:36:44.8487075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptl3cas5c/_remote_module_non_scriptable.py 2022-09-27T15:36:45.1094493Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:45.1104626Z 2022-09-27T15:36:45.1104726Z Running tests... 2022-09-27T15:36:45.1105280Z ---------------------------------------------------------------------- 2022-09-27T15:36:45.3543762Z test_async_function_simple (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22718 2022-09-27T15:36:45.3571368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22719 2022-09-27T15:36:45.3599767Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22720 2022-09-27T15:36:45.3628140Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22721 2022-09-27T15:36:46.4747551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:46.4748529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:46.4844330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:46.4844913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:46.4984078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:46.4984744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:46.5115620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:46.5116245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:46.5163839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:46.5164527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:46.5219294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:46.5219924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:46.5369950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:46.5370486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:46.5530657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:46.5531277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:46.6371660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdu4vqm9b 2022-09-27T15:36:46.6372461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdu4vqm9b/_remote_module_non_scriptable.py 2022-09-27T15:36:46.6482322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fo5uqs7 2022-09-27T15:36:46.6484110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fo5uqs7/_remote_module_non_scriptable.py 2022-09-27T15:36:46.6627971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaqr7ejjb 2022-09-27T15:36:46.6628940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaqr7ejjb/_remote_module_non_scriptable.py 2022-09-27T15:36:46.6783106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsr3971bv 2022-09-27T15:36:46.6784308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsr3971bv/_remote_module_non_scriptable.py 2022-09-27T15:36:46.9049211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:36:46.9114336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:36:46.9202644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:46.9354815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:36:47.6686163Z ok (2.558s) 2022-09-27T15:36:47.6686383Z 2022-09-27T15:36:47.6686701Z ---------------------------------------------------------------------- 2022-09-27T15:36:47.6687011Z Ran 1 test in 2.558s 2022-09-27T15:36:47.6687131Z 2022-09-27T15:36:47.6687179Z OK 2022-09-27T15:36:47.6687271Z 2022-09-27T15:36:47.6687363Z Generating XML reports... 2022-09-27T15:36:47.6723729Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153645.xml 2022-09-27T15:36:49.0539610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:49.0539997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:49.0547529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:49.0547883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:49.2150798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkepif8q 2022-09-27T15:36:49.2151893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkepif8q/_remote_module_non_scriptable.py 2022-09-27T15:36:49.4762278Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:49.4771590Z 2022-09-27T15:36:49.4771728Z Running tests... 2022-09-27T15:36:49.4772177Z ---------------------------------------------------------------------- 2022-09-27T15:36:49.7206364Z test_async_function_wrong_decorator_order (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22941 2022-09-27T15:36:49.7233203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22942 2022-09-27T15:36:49.7260975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22943 2022-09-27T15:36:49.7289948Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22944 2022-09-27T15:36:50.7958722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:50.7959392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:50.8313474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:50.8314199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:50.8555622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:50.8556231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:50.8582699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:50.8583367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:50.8684291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:50.8684809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:50.8948361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:50.8948982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:50.8973403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:50.8974023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:50.9086749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:50.9087364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:50.9608316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpck3vigs2 2022-09-27T15:36:50.9609110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpck3vigs2/_remote_module_non_scriptable.py 2022-09-27T15:36:51.0208446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxyr_qbhd 2022-09-27T15:36:51.0209243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxyr_qbhd/_remote_module_non_scriptable.py 2022-09-27T15:36:51.0239407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmn6u6qgh 2022-09-27T15:36:51.0241146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmn6u6qgh/_remote_module_non_scriptable.py 2022-09-27T15:36:51.0341648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppk0n6t06 2022-09-27T15:36:51.0343115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppk0n6t06/_remote_module_non_scriptable.py 2022-09-27T15:36:51.2198054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:51.2782623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:36:51.2804723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:36:51.2888368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:36:51.9346755Z ok (2.457s) 2022-09-27T15:36:51.9346989Z 2022-09-27T15:36:51.9347417Z ---------------------------------------------------------------------- 2022-09-27T15:36:51.9347831Z Ran 1 test in 2.457s 2022-09-27T15:36:51.9348012Z 2022-09-27T15:36:51.9348098Z OK 2022-09-27T15:36:51.9348241Z 2022-09-27T15:36:51.9348386Z Generating XML reports... 2022-09-27T15:36:51.9384357Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153649.xml 2022-09-27T15:36:53.3133274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:53.3133673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:53.3141602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:53.3141948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:53.4722341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7kk8nek 2022-09-27T15:36:53.4722817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7kk8nek/_remote_module_non_scriptable.py 2022-09-27T15:36:53.7331683Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:53.7341354Z 2022-09-27T15:36:53.7341470Z Running tests... 2022-09-27T15:36:53.7342009Z ---------------------------------------------------------------------- 2022-09-27T15:36:53.9790992Z test_async_function_wrong_return_type (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23164 2022-09-27T15:36:53.9817934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23165 2022-09-27T15:36:53.9846298Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23166 2022-09-27T15:36:53.9875752Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23167 2022-09-27T15:36:55.0849036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:55.0849694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:55.1198238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:55.1198974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:55.1209527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:55.1210126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:55.1555011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:55.1555691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:55.1601425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:55.1602303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:55.1960126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:55.1961113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:55.1972593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:55.1973259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:55.2329711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:55.2330349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:55.2493412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zne6vre 2022-09-27T15:36:55.2494382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zne6vre/_remote_module_non_scriptable.py 2022-09-27T15:36:55.2831887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xqrsp9z 2022-09-27T15:36:55.2832626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xqrsp9z/_remote_module_non_scriptable.py 2022-09-27T15:36:55.3240504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckoma3so 2022-09-27T15:36:55.3242482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckoma3so/_remote_module_non_scriptable.py 2022-09-27T15:36:55.3599475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8llxxnp 2022-09-27T15:36:55.3600149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8llxxnp/_remote_module_non_scriptable.py 2022-09-27T15:36:55.5101238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:55.5436183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:36:55.5841351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:36:55.6199105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:36:56.2932889Z ok (2.559s) 2022-09-27T15:36:56.2933183Z 2022-09-27T15:36:56.2933661Z ---------------------------------------------------------------------- 2022-09-27T15:36:56.2933913Z Ran 1 test in 2.559s 2022-09-27T15:36:56.2934028Z 2022-09-27T15:36:56.2934076Z OK 2022-09-27T15:36:56.2934166Z 2022-09-27T15:36:56.2934259Z Generating XML reports... 2022-09-27T15:36:56.2969856Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153653.xml 2022-09-27T15:36:57.6735583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:57.6735954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:57.6743656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:57.6744025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:57.8325660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobex1kjt 2022-09-27T15:36:57.8326807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobex1kjt/_remote_module_non_scriptable.py 2022-09-27T15:36:58.0929312Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:36:58.0938826Z 2022-09-27T15:36:58.0939000Z Running tests... 2022-09-27T15:36:58.0940528Z ---------------------------------------------------------------------- 2022-09-27T15:36:58.3391851Z test_async_function_wrong_return_type_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23387 2022-09-27T15:36:58.3418887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23388 2022-09-27T15:36:58.3446658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23389 2022-09-27T15:36:58.3475824Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23390 2022-09-27T15:36:59.4981533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:59.4982205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:59.5199707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:59.5200214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:59.5344747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:59.5345375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:59.5557113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:59.5557811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:59.5920655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:59.5921309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:59.6108713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:36:59.6109405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:36:59.6289409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:59.6290055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:59.6481233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:36:59.6481891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:36:59.6612306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozly0va7 2022-09-27T15:36:59.6613757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozly0va7/_remote_module_non_scriptable.py 2022-09-27T15:36:59.6834027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmlxdyl4d 2022-09-27T15:36:59.6835216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmlxdyl4d/_remote_module_non_scriptable.py 2022-09-27T15:36:59.7559007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzb1wb1zq 2022-09-27T15:36:59.7559772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzb1wb1zq/_remote_module_non_scriptable.py 2022-09-27T15:36:59.7750651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgboca3gv 2022-09-27T15:36:59.7751543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgboca3gv/_remote_module_non_scriptable.py 2022-09-27T15:36:59.9210492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:36:59.9446198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:00.0131563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:00.0344027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:00.6534062Z ok (2.559s) 2022-09-27T15:37:00.6534329Z 2022-09-27T15:37:00.6534797Z ---------------------------------------------------------------------- 2022-09-27T15:37:00.6535050Z Ran 1 test in 2.559s 2022-09-27T15:37:00.6535167Z 2022-09-27T15:37:00.6535227Z OK 2022-09-27T15:37:00.6535319Z 2022-09-27T15:37:00.6535415Z Generating XML reports... 2022-09-27T15:37:00.6571494Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153658.xml 2022-09-27T15:37:02.0406929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:02.0407295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:02.0415279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:02.0415638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:02.1991947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0lg66s2i 2022-09-27T15:37:02.1993080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0lg66s2i/_remote_module_non_scriptable.py 2022-09-27T15:37:02.4640880Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:02.4650263Z 2022-09-27T15:37:02.4650403Z Running tests... 2022-09-27T15:37:02.4650913Z ---------------------------------------------------------------------- 2022-09-27T15:37:02.7077296Z test_async_script_throw (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23610 2022-09-27T15:37:02.7104595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23611 2022-09-27T15:37:02.7132522Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23612 2022-09-27T15:37:02.7161535Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23613 2022-09-27T15:37:03.8690655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:03.8691366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:03.8857020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:03.8857677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:03.9044931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:03.9045671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:03.9198045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:03.9198697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:03.9213309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:03.9213992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:03.9331103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:03.9331757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:03.9549613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:03.9550222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:03.9698583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:03.9699220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:04.0310576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tw8fc63 2022-09-27T15:37:04.0312422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tw8fc63/_remote_module_non_scriptable.py 2022-09-27T15:37:04.0473110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7azx_lkr 2022-09-27T15:37:04.0474056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7azx_lkr/_remote_module_non_scriptable.py 2022-09-27T15:37:04.0812028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx7tbcizm 2022-09-27T15:37:04.0813371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx7tbcizm/_remote_module_non_scriptable.py 2022-09-27T15:37:04.0957254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp017yx9pz 2022-09-27T15:37:04.0959128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp017yx9pz/_remote_module_non_scriptable.py 2022-09-27T15:37:04.2886447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:37:04.3150969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:04.3398199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:04.3595324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:05.0219601Z ok (2.556s) 2022-09-27T15:37:05.0219873Z 2022-09-27T15:37:05.0220303Z ---------------------------------------------------------------------- 2022-09-27T15:37:05.0220559Z Ran 1 test in 2.557s 2022-09-27T15:37:05.0220674Z 2022-09-27T15:37:05.0220734Z OK 2022-09-27T15:37:05.0220826Z 2022-09-27T15:37:05.0220905Z Generating XML reports... 2022-09-27T15:37:05.0259109Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153702.xml 2022-09-27T15:37:06.3756305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:06.3756688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:06.3764589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:06.3765050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:06.5346777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwc5x37jb 2022-09-27T15:37:06.5347658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwc5x37jb/_remote_module_non_scriptable.py 2022-09-27T15:37:06.7942383Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:06.7952268Z 2022-09-27T15:37:06.7952505Z Running tests... 2022-09-27T15:37:06.7953112Z ---------------------------------------------------------------------- 2022-09-27T15:37:07.0352558Z test_async_script_udf (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23849 2022-09-27T15:37:07.0379330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23850 2022-09-27T15:37:07.0407460Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23851 2022-09-27T15:37:07.0435797Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23852 2022-09-27T15:37:08.1343438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:08.1344076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:08.1715713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:08.1716381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:08.2169637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:08.2170302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:08.2171103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:08.2171743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:08.2172811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:08.2173432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:08.2548901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:08.2549501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:08.2550255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:08.2550833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:08.2551612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:08.2552167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:08.2990402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0b_h8ay 2022-09-27T15:37:08.2991181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0b_h8ay/_remote_module_non_scriptable.py 2022-09-27T15:37:08.3816291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp15fjvxxr 2022-09-27T15:37:08.3817201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpbpdwtr 2022-09-27T15:37:08.3818078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp15fjvxxr/_remote_module_non_scriptable.py 2022-09-27T15:37:08.3819292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpbpdwtr/_remote_module_non_scriptable.py 2022-09-27T15:37:08.3820562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbptfco4v 2022-09-27T15:37:08.3822873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbptfco4v/_remote_module_non_scriptable.py 2022-09-27T15:37:08.5580222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:08.6385273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:37:08.6402836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:08.6576270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:09.3494271Z ok (2.554s) 2022-09-27T15:37:09.3494495Z 2022-09-27T15:37:09.3495029Z ---------------------------------------------------------------------- 2022-09-27T15:37:09.3495393Z Ran 1 test in 2.554s 2022-09-27T15:37:09.3495499Z 2022-09-27T15:37:09.3495559Z OK 2022-09-27T15:37:09.3495653Z 2022-09-27T15:37:09.3495747Z Generating XML reports... 2022-09-27T15:37:09.3531368Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153706.xml 2022-09-27T15:37:10.7344615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:10.7345302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:10.7353555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:10.7353937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:10.8904670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxplwjbnx 2022-09-27T15:37:10.8905460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxplwjbnx/_remote_module_non_scriptable.py 2022-09-27T15:37:11.1499040Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:11.1509137Z 2022-09-27T15:37:11.1509244Z Running tests... 2022-09-27T15:37:11.1509659Z ---------------------------------------------------------------------- 2022-09-27T15:37:11.3939029Z test_call_fork_in_jit_with_profiling (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24088 2022-09-27T15:37:11.3966318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24089 2022-09-27T15:37:11.3994169Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24090 2022-09-27T15:37:11.4022972Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24091 2022-09-27T15:37:12.5727033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:12.5727649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:12.5791639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:12.5792226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:12.5972959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:12.5973629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:12.6023929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:12.6024596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:12.6118189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:12.6118700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:12.6182092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:12.6182637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:12.6361271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:12.6361898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:12.6405712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:12.6406390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:12.7389802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddculj0v 2022-09-27T15:37:12.7390509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddculj0v/_remote_module_non_scriptable.py 2022-09-27T15:37:12.7453187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8bc76_d0 2022-09-27T15:37:12.7454146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8bc76_d0/_remote_module_non_scriptable.py 2022-09-27T15:37:12.7640152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwscjb_w 2022-09-27T15:37:12.7641197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwscjb_w/_remote_module_non_scriptable.py 2022-09-27T15:37:12.7666735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2x9vaf4 2022-09-27T15:37:12.7667718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2x9vaf4/_remote_module_non_scriptable.py 2022-09-27T15:37:12.9951336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:13.0024915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:37:13.0214667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:13.0222861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:13.4075474Z ok (2.256s) 2022-09-27T15:37:13.4075727Z 2022-09-27T15:37:13.4076272Z ---------------------------------------------------------------------- 2022-09-27T15:37:13.4076618Z Ran 1 test in 2.257s 2022-09-27T15:37:13.4076987Z 2022-09-27T15:37:13.4077053Z OK 2022-09-27T15:37:13.4077146Z 2022-09-27T15:37:13.4077242Z Generating XML reports... 2022-09-27T15:37:13.4112203Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153711.xml 2022-09-27T15:37:14.7974115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:14.7974514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:14.7982988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:14.7983350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:14.9545894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpooz4_6_l 2022-09-27T15:37:14.9547082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpooz4_6_l/_remote_module_non_scriptable.py 2022-09-27T15:37:15.2153454Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:15.2163459Z 2022-09-27T15:37:15.2163621Z Running tests... 2022-09-27T15:37:15.2164069Z ---------------------------------------------------------------------- 2022-09-27T15:37:15.4586523Z test_call_python_function_remotely_from_script_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24159 2022-09-27T15:37:15.4613516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24160 2022-09-27T15:37:15.4641279Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24161 2022-09-27T15:37:15.4670809Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24162 2022-09-27T15:37:16.5552642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:16.5553298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:16.5657072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:16.5657746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:16.5733935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:16.5734555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:16.5932752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:16.5933357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:16.6034132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:16.6034969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:16.6043879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:16.6044530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:16.6114215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:16.6115020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:16.6383182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:16.6383832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:16.7197248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyly8gbnd 2022-09-27T15:37:16.7198507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyly8gbnd/_remote_module_non_scriptable.py 2022-09-27T15:37:16.7336842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajcj0uba 2022-09-27T15:37:16.7337651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajcj0uba/_remote_module_non_scriptable.py 2022-09-27T15:37:16.7554093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97z61qfp 2022-09-27T15:37:16.7555115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97z61qfp/_remote_module_non_scriptable.py 2022-09-27T15:37:16.7668877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzl7qaxld 2022-09-27T15:37:16.7670311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzl7qaxld/_remote_module_non_scriptable.py 2022-09-27T15:37:16.9734669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:37:16.9888791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:17.0091147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:17.0214824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:17.6727812Z ok (2.456s) 2022-09-27T15:37:17.6728061Z 2022-09-27T15:37:17.6728587Z ---------------------------------------------------------------------- 2022-09-27T15:37:17.6728989Z Ran 1 test in 2.456s 2022-09-27T15:37:17.6729118Z 2022-09-27T15:37:17.6729167Z OK 2022-09-27T15:37:17.6729262Z 2022-09-27T15:37:17.6729359Z Generating XML reports... 2022-09-27T15:37:17.6764122Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153715.xml 2022-09-27T15:37:19.0608457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:19.0608874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:19.0616996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:19.0617364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:19.2194325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmeraold7 2022-09-27T15:37:19.2196017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmeraold7/_remote_module_non_scriptable.py 2022-09-27T15:37:19.4809814Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:19.4819692Z 2022-09-27T15:37:19.4819810Z Running tests... 2022-09-27T15:37:19.4820359Z ---------------------------------------------------------------------- 2022-09-27T15:37:19.7248516Z test_call_rpc_with_profiling (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24386 2022-09-27T15:37:19.7275935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24387 2022-09-27T15:37:19.7303248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24388 2022-09-27T15:37:19.7331212Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24389 2022-09-27T15:37:20.8357127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:20.8357782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:20.8599770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:20.8600321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:20.8733279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:20.8733888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:20.8891593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:20.8892268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:20.8912472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:20.8913030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:20.8995068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:20.8995611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:20.9275457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:20.9276252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:20.9297706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:20.9298312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:21.0006669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6ughayn 2022-09-27T15:37:21.0007879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6ughayn/_remote_module_non_scriptable.py 2022-09-27T15:37:21.0271204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1n3wus0 2022-09-27T15:37:21.0272044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1n3wus0/_remote_module_non_scriptable.py 2022-09-27T15:37:21.0549817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mf9f3n8 2022-09-27T15:37:21.0551108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mf9f3n8/_remote_module_non_scriptable.py 2022-09-27T15:37:21.0559783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_59zmju 2022-09-27T15:37:21.0561941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_59zmju/_remote_module_non_scriptable.py 2022-09-27T15:37:21.2613970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:21.2875673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:21.3116171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:21.3152069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:37:22.0391007Z ok (2.557s) 2022-09-27T15:37:22.0391232Z 2022-09-27T15:37:22.0391801Z ---------------------------------------------------------------------- 2022-09-27T15:37:22.0392144Z Ran 1 test in 2.557s 2022-09-27T15:37:22.0392264Z 2022-09-27T15:37:22.0392325Z OK 2022-09-27T15:37:22.0392419Z 2022-09-27T15:37:22.0392500Z Generating XML reports... 2022-09-27T15:37:22.0429481Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153719.xml 2022-09-27T15:37:23.4176776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:23.4177194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:23.4184753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:23.4185115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:23.5746933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60141wwl 2022-09-27T15:37:23.5747925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60141wwl/_remote_module_non_scriptable.py 2022-09-27T15:37:23.8356326Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:23.8366065Z 2022-09-27T15:37:23.8366204Z Running tests... 2022-09-27T15:37:23.8367033Z ---------------------------------------------------------------------- 2022-09-27T15:37:24.0820013Z test_call_script_function_that_not_exists_remotely_from_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24613 2022-09-27T15:37:24.0847408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24614 2022-09-27T15:37:24.0874472Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24615 2022-09-27T15:37:24.0904414Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24616 2022-09-27T15:37:25.2500070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:25.2500724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:25.2725257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:25.2725952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:25.2876738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:25.2877276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:25.2896841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:25.2897363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:25.3122952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:25.3123517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:25.3253816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:25.3254477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:25.3376394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:25.3377034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:25.3754077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:25.3754906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:25.4171671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpijqp3jac 2022-09-27T15:37:25.4172376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpijqp3jac/_remote_module_non_scriptable.py 2022-09-27T15:37:25.4400768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0vp99d6 2022-09-27T15:37:25.4401587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0vp99d6/_remote_module_non_scriptable.py 2022-09-27T15:37:25.4532108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeaty6cas 2022-09-27T15:37:25.4533379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeaty6cas/_remote_module_non_scriptable.py 2022-09-27T15:37:25.5020633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2nki1k3 2022-09-27T15:37:25.5021387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2nki1k3/_remote_module_non_scriptable.py 2022-09-27T15:37:25.6775165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:25.6996071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:37:25.7146688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:25.7651548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:26.4961076Z ok (2.659s) 2022-09-27T15:37:26.4961353Z 2022-09-27T15:37:26.4961677Z ---------------------------------------------------------------------- 2022-09-27T15:37:26.4962011Z Ran 1 test in 2.659s 2022-09-27T15:37:26.4962130Z 2022-09-27T15:37:26.4962177Z OK 2022-09-27T15:37:26.4962270Z 2022-09-27T15:37:26.4962367Z Generating XML reports... 2022-09-27T15:37:26.4996789Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153723.xml 2022-09-27T15:37:27.8864391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:27.8865038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:27.8874465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:27.8875362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:28.0523353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbok900c 2022-09-27T15:37:28.0524022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbok900c/_remote_module_non_scriptable.py 2022-09-27T15:37:28.3135595Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:28.3146079Z 2022-09-27T15:37:28.3146399Z Running tests... 2022-09-27T15:37:28.3146982Z ---------------------------------------------------------------------- 2022-09-27T15:37:28.5599541Z test_call_script_function_that_raises_remotely_from_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24840 2022-09-27T15:37:28.5627040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24841 2022-09-27T15:37:28.5654595Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24842 2022-09-27T15:37:28.5684329Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24843 2022-09-27T15:37:29.7224314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:29.7225015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:29.7530734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:29.7531477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:29.7574040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:29.7574665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:29.7861278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:29.7861891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:29.7898112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:29.7898764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:29.8001616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:29.8002238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:29.8233401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:29.8234022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:29.8370498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:29.8371198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:29.8845503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe64rhwc1 2022-09-27T15:37:29.8846324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe64rhwc1/_remote_module_non_scriptable.py 2022-09-27T15:37:29.9159045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptmg4gf9x 2022-09-27T15:37:29.9159870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptmg4gf9x/_remote_module_non_scriptable.py 2022-09-27T15:37:29.9498013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgg7ffbx0 2022-09-27T15:37:29.9498828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgg7ffbx0/_remote_module_non_scriptable.py 2022-09-27T15:37:29.9634878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64xad5pp 2022-09-27T15:37:29.9635755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64xad5pp/_remote_module_non_scriptable.py 2022-09-27T15:37:30.1418208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:30.1728102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:37:30.2062312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:30.2218145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:30.9743526Z ok (2.659s) 2022-09-27T15:37:30.9743749Z 2022-09-27T15:37:30.9744084Z ---------------------------------------------------------------------- 2022-09-27T15:37:30.9744323Z Ran 1 test in 2.660s 2022-09-27T15:37:30.9744445Z 2022-09-27T15:37:30.9744507Z OK 2022-09-27T15:37:30.9744603Z 2022-09-27T15:37:30.9744697Z Generating XML reports... 2022-09-27T15:37:30.9780628Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153728.xml 2022-09-27T15:37:32.3473858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:32.3474243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:32.3482385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:32.3482753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:32.5043527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_rnjyitk 2022-09-27T15:37:32.5045081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_rnjyitk/_remote_module_non_scriptable.py 2022-09-27T15:37:32.7648748Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:32.7658087Z 2022-09-27T15:37:32.7658199Z Running tests... 2022-09-27T15:37:32.7658597Z ---------------------------------------------------------------------- 2022-09-27T15:37:33.0119718Z test_callback_chain (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25067 2022-09-27T15:37:33.0146930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25068 2022-09-27T15:37:33.0174533Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25069 2022-09-27T15:37:33.0204671Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25070 2022-09-27T15:37:34.1353177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:34.1353814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:34.1649463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:34.1650144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:34.1704921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:34.1705485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:34.1744996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:34.1745516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:34.2022391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:34.2023050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:34.2114342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:34.2115161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:34.2297326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:34.2298011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:34.2650177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:34.2650816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:34.2988927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkecenio6 2022-09-27T15:37:34.2989889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkecenio6/_remote_module_non_scriptable.py 2022-09-27T15:37:34.3290768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp076a541n 2022-09-27T15:37:34.3291478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp076a541n/_remote_module_non_scriptable.py 2022-09-27T15:37:34.3566545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwp4tdabm 2022-09-27T15:37:34.3567448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwp4tdabm/_remote_module_non_scriptable.py 2022-09-27T15:37:34.3938134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpuv027k 2022-09-27T15:37:34.3938911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpuv027k/_remote_module_non_scriptable.py 2022-09-27T15:37:34.5536803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:34.5841018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:37:34.6106993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:34.6499392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:35.2261357Z ok (2.460s) 2022-09-27T15:37:35.2261589Z 2022-09-27T15:37:35.2262108Z ---------------------------------------------------------------------- 2022-09-27T15:37:35.2262525Z Ran 1 test in 2.460s 2022-09-27T15:37:35.2262630Z 2022-09-27T15:37:35.2262701Z OK 2022-09-27T15:37:35.2262793Z 2022-09-27T15:37:35.2262891Z Generating XML reports... 2022-09-27T15:37:35.2300065Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153732.xml 2022-09-27T15:37:36.5909473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:36.5909851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:36.5917968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:36.5918468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:36.7515926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpram3dpv7 2022-09-27T15:37:36.7516547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpram3dpv7/_remote_module_non_scriptable.py 2022-09-27T15:37:37.0153330Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:37.0162825Z 2022-09-27T15:37:37.0162916Z Running tests... 2022-09-27T15:37:37.0163378Z ---------------------------------------------------------------------- 2022-09-27T15:37:37.2536306Z test_callback_simple (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25290 2022-09-27T15:37:37.2563992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25291 2022-09-27T15:37:37.2590714Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25292 2022-09-27T15:37:37.2619809Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25293 2022-09-27T15:37:38.4043326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:38.4044027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:38.4404890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:38.4405571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:38.4430471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:38.4431110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:38.4650691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:38.4651356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:38.4795766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:38.4796547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:38.5006527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:38.5007236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:38.5016948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:38.5017548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:38.5375209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:38.5375887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:38.5681904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7h2_tqgn 2022-09-27T15:37:38.5685855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7h2_tqgn/_remote_module_non_scriptable.py 2022-09-27T15:37:38.6064409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdy9lp96a 2022-09-27T15:37:38.6065110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdy9lp96a/_remote_module_non_scriptable.py 2022-09-27T15:37:38.6284267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa581ylmr 2022-09-27T15:37:38.6285557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa581ylmr/_remote_module_non_scriptable.py 2022-09-27T15:37:38.6649777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpey584sl5 2022-09-27T15:37:38.6650723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpey584sl5/_remote_module_non_scriptable.py 2022-09-27T15:37:38.8250782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:38.8659742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:38.8860421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:37:38.9247311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:39.5680970Z ok (2.551s) 2022-09-27T15:37:39.5681142Z 2022-09-27T15:37:39.5681615Z ---------------------------------------------------------------------- 2022-09-27T15:37:39.5682053Z Ran 1 test in 2.552s 2022-09-27T15:37:39.5682271Z 2022-09-27T15:37:39.5682373Z OK 2022-09-27T15:37:39.5682481Z 2022-09-27T15:37:39.5682574Z Generating XML reports... 2022-09-27T15:37:39.5717928Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153737.xml 2022-09-27T15:37:40.9442694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:40.9443098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:40.9450783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:40.9451153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:41.0985850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7nfea9yc 2022-09-27T15:37:41.0987383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7nfea9yc/_remote_module_non_scriptable.py 2022-09-27T15:37:41.3578037Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:41.3587653Z 2022-09-27T15:37:41.3588082Z Running tests... 2022-09-27T15:37:41.3588483Z ---------------------------------------------------------------------- 2022-09-27T15:37:41.6006149Z test_callback_with_exception (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25529 2022-09-27T15:37:41.6034837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25530 2022-09-27T15:37:41.6062537Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25531 2022-09-27T15:37:41.6098332Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25532 2022-09-27T15:37:42.7009550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:42.7010147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:42.7066251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:42.7067209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:42.7072110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:42.7072737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:42.7101431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:42.7102090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:42.7392175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:42.7392939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:42.7452636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:42.7453260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:42.7459255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:42.7459884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:42.7489836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:42.7490726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:42.8657508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4uc495zh 2022-09-27T15:37:42.8659184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4uc495zh/_remote_module_non_scriptable.py 2022-09-27T15:37:42.8712937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc60q79ws 2022-09-27T15:37:42.8714811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc60q79ws/_remote_module_non_scriptable.py 2022-09-27T15:37:42.8718731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinqy6ic5 2022-09-27T15:37:42.8720669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinqy6ic5/_remote_module_non_scriptable.py 2022-09-27T15:37:42.8745122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphq1a02u1 2022-09-27T15:37:42.8747521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphq1a02u1/_remote_module_non_scriptable.py 2022-09-27T15:37:43.1206157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:43.1256489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:37:43.1281841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:43.1305770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:43.9156759Z ok (2.556s) 2022-09-27T15:37:43.9156968Z 2022-09-27T15:37:43.9157395Z ---------------------------------------------------------------------- 2022-09-27T15:37:43.9157738Z Ran 1 test in 2.557s 2022-09-27T15:37:43.9157946Z 2022-09-27T15:37:43.9158063Z OK 2022-09-27T15:37:43.9158234Z 2022-09-27T15:37:43.9158409Z Generating XML reports... 2022-09-27T15:37:43.9192793Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153741.xml 2022-09-27T15:37:45.2815445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:45.2815838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:45.2823978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:45.2824354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:45.4379054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9c4aehy0 2022-09-27T15:37:45.4379723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9c4aehy0/_remote_module_non_scriptable.py 2022-09-27T15:37:45.7028487Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:45.7039223Z 2022-09-27T15:37:45.7039480Z Running tests... 2022-09-27T15:37:45.7040168Z ---------------------------------------------------------------------- 2022-09-27T15:37:45.9440947Z test_create_local_script_class_rref_in_py (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25768 2022-09-27T15:37:45.9468252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25769 2022-09-27T15:37:45.9495818Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25770 2022-09-27T15:37:45.9525234Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25771 2022-09-27T15:37:47.0077701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:47.0078325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:47.0358788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:47.0359652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:47.0447788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:47.0448808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:47.0602980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:47.0603640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:47.0733112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:47.0733781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:47.0766887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:47.0767526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:47.1002554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:47.1003235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:47.1156063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:47.1156770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:47.1743942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7cvzmkeq 2022-09-27T15:37:47.1744714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7cvzmkeq/_remote_module_non_scriptable.py 2022-09-27T15:37:47.2015583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzi7cv_q 2022-09-27T15:37:47.2016414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzi7cv_q/_remote_module_non_scriptable.py 2022-09-27T15:37:47.2286338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggtvqlmc 2022-09-27T15:37:47.2287256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggtvqlmc/_remote_module_non_scriptable.py 2022-09-27T15:37:47.2430934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01n2pwim 2022-09-27T15:37:47.2432504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01n2pwim/_remote_module_non_scriptable.py 2022-09-27T15:37:47.4378730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:47.4621220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:47.4869740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:47.5028799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:37:48.1581963Z ok (2.454s) 2022-09-27T15:37:48.1582188Z 2022-09-27T15:37:48.1582660Z ---------------------------------------------------------------------- 2022-09-27T15:37:48.1583036Z Ran 1 test in 2.454s 2022-09-27T15:37:48.1583247Z 2022-09-27T15:37:48.1583345Z OK 2022-09-27T15:37:48.1583484Z 2022-09-27T15:37:48.1583620Z Generating XML reports... 2022-09-27T15:37:48.1619826Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153745.xml 2022-09-27T15:37:49.5320177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:49.5320559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:49.5328618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:49.5328977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:49.6936652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpai3z3v9k 2022-09-27T15:37:49.6937587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpai3z3v9k/_remote_module_non_scriptable.py 2022-09-27T15:37:49.9536234Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:49.9545713Z 2022-09-27T15:37:49.9545840Z Running tests... 2022-09-27T15:37:49.9546250Z ---------------------------------------------------------------------- 2022-09-27T15:37:50.1962755Z test_create_local_script_module_rref_in_py (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25991 2022-09-27T15:37:50.1990024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25992 2022-09-27T15:37:50.2017597Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25993 2022-09-27T15:37:50.2046770Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25994 2022-09-27T15:37:51.3314316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:51.3315239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:51.3673661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:51.3674285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:51.3930196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:51.3930984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:51.3931769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:51.3932398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:51.4114958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:51.4115508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:51.4328761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:51.4329404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:51.4330120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:51.4330664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:51.4510525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:51.4511157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:51.4962815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp605vwgdk 2022-09-27T15:37:51.4963656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp605vwgdk/_remote_module_non_scriptable.py 2022-09-27T15:37:51.5603801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ugs5npc 2022-09-27T15:37:51.5604978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ugs5npc/_remote_module_non_scriptable.py 2022-09-27T15:37:51.5605659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyubmckx_ 2022-09-27T15:37:51.5608120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyubmckx_/_remote_module_non_scriptable.py 2022-09-27T15:37:51.5791032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6n1r_ml 2022-09-27T15:37:51.5791916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6n1r_ml/_remote_module_non_scriptable.py 2022-09-27T15:37:51.7546051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:51.8221090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:51.8221768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:37:51.8404802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:52.5125989Z ok (2.558s) 2022-09-27T15:37:52.5126275Z 2022-09-27T15:37:52.5126786Z ---------------------------------------------------------------------- 2022-09-27T15:37:52.5127254Z Ran 1 test in 2.558s 2022-09-27T15:37:52.5127467Z 2022-09-27T15:37:52.5127577Z OK 2022-09-27T15:37:52.5127727Z 2022-09-27T15:37:52.5127885Z Generating XML reports... 2022-09-27T15:37:52.5162129Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153749.xml 2022-09-27T15:37:53.8739516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:53.8739907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:53.8747612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:53.8748142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:54.0293113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwmxv0ubi 2022-09-27T15:37:54.0294084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwmxv0ubi/_remote_module_non_scriptable.py 2022-09-27T15:37:54.2901871Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:54.2912105Z 2022-09-27T15:37:54.2912517Z Running tests... 2022-09-27T15:37:54.2912935Z ---------------------------------------------------------------------- 2022-09-27T15:37:54.5387803Z test_create_script_module_on_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26214 2022-09-27T15:37:54.5415893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26215 2022-09-27T15:37:54.5443091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26216 2022-09-27T15:37:54.5472215Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26217 2022-09-27T15:37:55.6578380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:55.6579050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:55.6609044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:55.6609688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:55.6699035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:55.6699542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:55.6733173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:55.6733686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:55.6955410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:55.6956025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:55.6991454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:55.6992041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:55.7079943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:55.7080975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:55.7105936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:55.7106743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:55.8209665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppj5_9swx 2022-09-27T15:37:55.8210334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppj5_9swx/_remote_module_non_scriptable.py 2022-09-27T15:37:55.8247321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtrn88ms 2022-09-27T15:37:55.8248669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtrn88ms/_remote_module_non_scriptable.py 2022-09-27T15:37:55.8334107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3h279b7b 2022-09-27T15:37:55.8335650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3h279b7b/_remote_module_non_scriptable.py 2022-09-27T15:37:55.8357000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14vnjqt1 2022-09-27T15:37:55.8358029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14vnjqt1/_remote_module_non_scriptable.py 2022-09-27T15:37:56.0840946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:37:56.0906313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:37:56.0936988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:37:56.0942528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:37:56.8531103Z ok (2.561s) 2022-09-27T15:37:56.8531352Z 2022-09-27T15:37:56.8531856Z ---------------------------------------------------------------------- 2022-09-27T15:37:56.8532324Z Ran 1 test in 2.562s 2022-09-27T15:37:56.8532528Z 2022-09-27T15:37:56.8532620Z OK 2022-09-27T15:37:56.8532789Z 2022-09-27T15:37:56.8532974Z Generating XML reports... 2022-09-27T15:37:56.8567971Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153754.xml 2022-09-27T15:37:58.2222413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:37:58.2223097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:37:58.2232107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:37:58.2232680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:37:58.3845532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xmn1ftb 2022-09-27T15:37:58.3846263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xmn1ftb/_remote_module_non_scriptable.py 2022-09-27T15:37:58.6472960Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:37:58.6483412Z 2022-09-27T15:37:58.6483740Z Running tests... 2022-09-27T15:37:58.6484344Z ---------------------------------------------------------------------- 2022-09-27T15:37:58.8877782Z test_future_passed_between_python_and_jit (__main__.TensorPipeJitRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84046 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.239s) 2022-09-27T15:37:58.8878697Z 2022-09-27T15:37:58.8879067Z ---------------------------------------------------------------------- 2022-09-27T15:37:58.8879490Z Ran 1 test in 0.239s 2022-09-27T15:37:58.8879683Z 2022-09-27T15:37:58.8879788Z OK (skipped=1) 2022-09-27T15:37:58.8879975Z 2022-09-27T15:37:58.8880118Z Generating XML reports... 2022-09-27T15:37:58.8903141Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153758.xml 2022-09-27T15:38:00.1866161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:00.1866810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:00.1875391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:00.1876007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:00.3436245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_cdwhhna 2022-09-27T15:38:00.3436922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_cdwhhna/_remote_module_non_scriptable.py 2022-09-27T15:38:00.6044225Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:00.6054377Z 2022-09-27T15:38:00.6054693Z Running tests... 2022-09-27T15:38:00.6055319Z ---------------------------------------------------------------------- 2022-09-27T15:38:00.8504376Z test_future_python_annotation (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26447 2022-09-27T15:38:00.8532122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26448 2022-09-27T15:38:00.8560480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26449 2022-09-27T15:38:00.8588866Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26450 2022-09-27T15:38:01.9588414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:01.9589062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:01.9708478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:01.9709144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:01.9982364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:01.9983062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:02.0010189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:02.0010801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:02.0026359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:02.0026963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:02.0102819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:02.0103516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:02.0405195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:02.0405805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:02.0414488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:02.0415057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:02.1263528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0uhied4 2022-09-27T15:38:02.1264750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0uhied4/_remote_module_non_scriptable.py 2022-09-27T15:38:02.1382556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzhealim 2022-09-27T15:38:02.1383751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzhealim/_remote_module_non_scriptable.py 2022-09-27T15:38:02.1689454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_npsgbz 2022-09-27T15:38:02.1690746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_npsgbz/_remote_module_non_scriptable.py 2022-09-27T15:38:02.1691843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcp1uujzj 2022-09-27T15:38:02.1695848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcp1uujzj/_remote_module_non_scriptable.py 2022-09-27T15:38:02.3965322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:02.4017425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:02.4260907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:02.4293299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:38:03.0645551Z ok (2.459s) 2022-09-27T15:38:03.0645791Z 2022-09-27T15:38:03.0646246Z ---------------------------------------------------------------------- 2022-09-27T15:38:03.0646525Z Ran 1 test in 2.459s 2022-09-27T15:38:03.0646643Z 2022-09-27T15:38:03.0646705Z OK 2022-09-27T15:38:03.0646784Z 2022-09-27T15:38:03.0646880Z Generating XML reports... 2022-09-27T15:38:03.0682147Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153800.xml 2022-09-27T15:38:04.4391927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:04.4392319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:04.4400377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:04.4400742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:04.5946982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9k7uwv1 2022-09-27T15:38:04.5948198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9k7uwv1/_remote_module_non_scriptable.py 2022-09-27T15:38:04.8535041Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:04.8544947Z 2022-09-27T15:38:04.8545461Z Running tests... 2022-09-27T15:38:04.8546057Z ---------------------------------------------------------------------- 2022-09-27T15:38:05.0954914Z test_kwargs_not_passed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26674 2022-09-27T15:38:05.0981636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26675 2022-09-27T15:38:05.1009344Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26676 2022-09-27T15:38:05.1037807Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26677 2022-09-27T15:38:06.1695406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:06.1696035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:06.2049206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:06.2049842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:06.2223142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:06.2223752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:06.2258660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:06.2259281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:06.2284726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:06.2285383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:06.2602390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:06.2603026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:06.2639357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:06.2640019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:06.2657478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:06.2658111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:06.3319921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcvgju5oi 2022-09-27T15:38:06.3320685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcvgju5oi/_remote_module_non_scriptable.py 2022-09-27T15:38:06.3867715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzrjw_ch 2022-09-27T15:38:06.3868614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzrjw_ch/_remote_module_non_scriptable.py 2022-09-27T15:38:06.3951982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp053su6l3 2022-09-27T15:38:06.3953367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp053su6l3/_remote_module_non_scriptable.py 2022-09-27T15:38:06.4083831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_ysfj_f 2022-09-27T15:38:06.4085569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_ysfj_f/_remote_module_non_scriptable.py 2022-09-27T15:38:06.5897110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:06.6429386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:06.6488439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:38:06.6695287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:07.3095120Z ok (2.454s) 2022-09-27T15:38:07.3095344Z 2022-09-27T15:38:07.3095734Z ---------------------------------------------------------------------- 2022-09-27T15:38:07.3096128Z Ran 1 test in 2.455s 2022-09-27T15:38:07.3096324Z 2022-09-27T15:38:07.3096426Z OK 2022-09-27T15:38:07.3096556Z 2022-09-27T15:38:07.3096695Z Generating XML reports... 2022-09-27T15:38:07.3133261Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153804.xml 2022-09-27T15:38:08.7253128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:08.7253520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:08.7261528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:08.7261889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:08.8820857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxs_2eafs 2022-09-27T15:38:08.8821725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxs_2eafs/_remote_module_non_scriptable.py 2022-09-27T15:38:09.1420380Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:09.1429813Z 2022-09-27T15:38:09.1429951Z Running tests... 2022-09-27T15:38:09.1430632Z ---------------------------------------------------------------------- 2022-09-27T15:38:09.3868079Z test_less_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26901 2022-09-27T15:38:09.3895580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26902 2022-09-27T15:38:09.3923864Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26903 2022-09-27T15:38:09.3953050Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26904 2022-09-27T15:38:10.5230984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:10.5231739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:10.5404552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:10.5405209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:10.5583333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:10.5583982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:10.5692319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:10.5692970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:10.5693778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:10.5694416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:10.5759344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:10.5759895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:10.6058785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:10.6059416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:10.6063625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:10.6064192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:10.6853008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptsc3ntbs 2022-09-27T15:38:10.6853813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptsc3ntbs/_remote_module_non_scriptable.py 2022-09-27T15:38:10.7023365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9v6os9t5 2022-09-27T15:38:10.7024141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9v6os9t5/_remote_module_non_scriptable.py 2022-09-27T15:38:10.7312778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qr67las 2022-09-27T15:38:10.7314443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qr67las/_remote_module_non_scriptable.py 2022-09-27T15:38:10.7317157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_ih01uc 2022-09-27T15:38:10.7319400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_ih01uc/_remote_module_non_scriptable.py 2022-09-27T15:38:10.9423295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:10.9579830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:10.9872477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:10.9873064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:38:11.6008215Z ok (2.457s) 2022-09-27T15:38:11.6008810Z 2022-09-27T15:38:11.6009429Z ---------------------------------------------------------------------- 2022-09-27T15:38:11.6009791Z Ran 1 test in 2.458s 2022-09-27T15:38:11.6010117Z 2022-09-27T15:38:11.6010184Z OK 2022-09-27T15:38:11.6010278Z 2022-09-27T15:38:11.6010373Z Generating XML reports... 2022-09-27T15:38:11.6044841Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153809.xml 2022-09-27T15:38:12.9587782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:12.9588240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:12.9596488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:12.9596846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:13.1182725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1ivmcne 2022-09-27T15:38:13.1183825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1ivmcne/_remote_module_non_scriptable.py 2022-09-27T15:38:13.3810287Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:13.3819904Z 2022-09-27T15:38:13.3820018Z Running tests... 2022-09-27T15:38:13.3820472Z ---------------------------------------------------------------------- 2022-09-27T15:38:13.6312729Z test_load_script_module_with_pickled_rref (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27124 2022-09-27T15:38:13.6340250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27125 2022-09-27T15:38:13.6367759Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27126 2022-09-27T15:38:13.6396742Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27127 2022-09-27T15:38:14.7598337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:14.7599003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:14.7679712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:14.7680399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:14.7973268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:14.7973953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:14.8037680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:14.8038314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:14.8058385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:14.8059101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:14.8064622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:14.8065279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:14.8422773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:14.8423536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:14.8454143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:14.8454773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:14.9241082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvv37u92w 2022-09-27T15:38:14.9241881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvv37u92w/_remote_module_non_scriptable.py 2022-09-27T15:38:14.9334273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmo_p2ai 2022-09-27T15:38:14.9335423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmo_p2ai/_remote_module_non_scriptable.py 2022-09-27T15:38:14.9688186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthmauugd 2022-09-27T15:38:14.9688901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthmauugd/_remote_module_non_scriptable.py 2022-09-27T15:38:14.9715063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp327kemg_ 2022-09-27T15:38:14.9716483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp327kemg_/_remote_module_non_scriptable.py 2022-09-27T15:38:15.1781298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:15.1887021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:38:15.2244461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:15.2281512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:15.9456486Z ok (2.563s) 2022-09-27T15:38:15.9456708Z 2022-09-27T15:38:15.9457015Z ---------------------------------------------------------------------- 2022-09-27T15:38:15.9457270Z Ran 1 test in 2.564s 2022-09-27T15:38:15.9457389Z 2022-09-27T15:38:15.9457453Z OK 2022-09-27T15:38:15.9457546Z 2022-09-27T15:38:15.9457642Z Generating XML reports... 2022-09-27T15:38:15.9495340Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153813.xml 2022-09-27T15:38:17.3407399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:17.3417447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:17.3418215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:17.3418924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:17.4999323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgkzbuf69 2022-09-27T15:38:17.4999809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgkzbuf69/_remote_module_non_scriptable.py 2022-09-27T15:38:17.7599852Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:17.7609959Z 2022-09-27T15:38:17.7610072Z Running tests... 2022-09-27T15:38:17.7610497Z ---------------------------------------------------------------------- 2022-09-27T15:38:18.0039110Z test_local_rref_local_value (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27347 2022-09-27T15:38:18.0065838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27348 2022-09-27T15:38:18.0093636Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27349 2022-09-27T15:38:18.0122975Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27350 2022-09-27T15:38:19.1569278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:19.1569956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:19.1869771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:19.1870393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:19.1960403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:19.1961025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:19.2270245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:19.2270934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:19.2369752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:19.2370384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:19.2378488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:19.2379067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:19.2755206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:19.2755897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:19.2759536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:19.2760228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:19.3236643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvozkzfw5 2022-09-27T15:38:19.3237406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvozkzfw5/_remote_module_non_scriptable.py 2022-09-27T15:38:19.3535590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpukj962ql 2022-09-27T15:38:19.3536665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpukj962ql/_remote_module_non_scriptable.py 2022-09-27T15:38:19.4025927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpztn_jkfx 2022-09-27T15:38:19.4026657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpztn_jkfx/_remote_module_non_scriptable.py 2022-09-27T15:38:19.4039744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzox448sn 2022-09-27T15:38:19.4041551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzox448sn/_remote_module_non_scriptable.py 2022-09-27T15:38:19.5860022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:19.6133018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:19.6620550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:19.6661807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:38:20.4183597Z ok (2.657s) 2022-09-27T15:38:20.4183924Z 2022-09-27T15:38:20.4184432Z ---------------------------------------------------------------------- 2022-09-27T15:38:20.4184692Z Ran 1 test in 2.657s 2022-09-27T15:38:20.4184811Z 2022-09-27T15:38:20.4184873Z OK 2022-09-27T15:38:20.4184952Z 2022-09-27T15:38:20.4185050Z Generating XML reports... 2022-09-27T15:38:20.4219169Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153817.xml 2022-09-27T15:38:21.7749881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:21.7750260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:21.7758793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:21.7759156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:21.9349895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fcn2vr2 2022-09-27T15:38:21.9350662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fcn2vr2/_remote_module_non_scriptable.py 2022-09-27T15:38:22.1946284Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:22.1956441Z 2022-09-27T15:38:22.1956542Z Running tests... 2022-09-27T15:38:22.1957424Z ---------------------------------------------------------------------- 2022-09-27T15:38:22.4397373Z test_more_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27570 2022-09-27T15:38:22.4424668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27571 2022-09-27T15:38:22.4451881Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27572 2022-09-27T15:38:22.4480860Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27573 2022-09-27T15:38:23.5679636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:23.5680251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:23.5856508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:23.5857225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:23.6042096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:23.6042663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:23.6074311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:23.6075018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:23.6222910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:23.6223499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:23.6305040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:23.6305668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:23.6438724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:23.6439422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:23.6663607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:23.6664254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:23.7312278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfegu3eia 2022-09-27T15:38:23.7313571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfegu3eia/_remote_module_non_scriptable.py 2022-09-27T15:38:23.7488422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpibqxuyi6 2022-09-27T15:38:23.7489368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpibqxuyi6/_remote_module_non_scriptable.py 2022-09-27T15:38:23.7690726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25q6s6i4 2022-09-27T15:38:23.7691941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25q6s6i4/_remote_module_non_scriptable.py 2022-09-27T15:38:23.7930400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe34f42qm 2022-09-27T15:38:23.7931192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe34f42qm/_remote_module_non_scriptable.py 2022-09-27T15:38:23.9897609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:38:24.0053265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:24.0452420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:24.0540974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:24.6536988Z ok (2.458s) 2022-09-27T15:38:24.6537413Z 2022-09-27T15:38:24.6537712Z ---------------------------------------------------------------------- 2022-09-27T15:38:24.6538032Z Ran 1 test in 2.458s 2022-09-27T15:38:24.6538153Z 2022-09-27T15:38:24.6538200Z OK 2022-09-27T15:38:24.6538294Z 2022-09-27T15:38:24.6538392Z Generating XML reports... 2022-09-27T15:38:24.6574895Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153822.xml 2022-09-27T15:38:26.0367507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:26.0367895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:26.0376407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:26.0376750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:26.1989235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphew8tjnx 2022-09-27T15:38:26.1990453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphew8tjnx/_remote_module_non_scriptable.py 2022-09-27T15:38:26.4635500Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:26.4645317Z 2022-09-27T15:38:26.4645459Z Running tests... 2022-09-27T15:38:26.4645873Z ---------------------------------------------------------------------- 2022-09-27T15:38:26.7052521Z test_my_script_module_with_rrefs (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27793 2022-09-27T15:38:26.7079220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27794 2022-09-27T15:38:26.7107038Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27795 2022-09-27T15:38:26.7136163Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27796 2022-09-27T15:38:27.7934253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:27.7934953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:27.8297967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:27.8298611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:27.8304744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:27.8305348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:27.8411509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:27.8412187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:27.8427281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:27.8427937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:27.8690973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:27.8691638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:27.8795511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:27.8796147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:27.8812798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:27.8813419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:27.9579036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmmy5_ke 2022-09-27T15:38:27.9580208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmmy5_ke/_remote_module_non_scriptable.py 2022-09-27T15:38:27.9964735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppbxo7m1v 2022-09-27T15:38:27.9965824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppbxo7m1v/_remote_module_non_scriptable.py 2022-09-27T15:38:28.0078232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ub6h4vo 2022-09-27T15:38:28.0078983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ub6h4vo/_remote_module_non_scriptable.py 2022-09-27T15:38:28.0097402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbfxmyvex 2022-09-27T15:38:28.0099547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbfxmyvex/_remote_module_non_scriptable.py 2022-09-27T15:38:28.2179925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:28.2562094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:28.2690371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:38:28.2879888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:29.0195626Z ok (2.555s) 2022-09-27T15:38:29.0195858Z 2022-09-27T15:38:29.0196314Z ---------------------------------------------------------------------- 2022-09-27T15:38:29.0196699Z Ran 1 test in 2.555s 2022-09-27T15:38:29.0196875Z 2022-09-27T15:38:29.0196974Z OK 2022-09-27T15:38:29.0197133Z 2022-09-27T15:38:29.0197272Z Generating XML reports... 2022-09-27T15:38:29.0232974Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153826.xml 2022-09-27T15:38:30.3949962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:30.3950609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:30.3958949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:30.3959562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:30.5522936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48arkc9x 2022-09-27T15:38:30.5523988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48arkc9x/_remote_module_non_scriptable.py 2022-09-27T15:38:30.8140405Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:30.8149980Z 2022-09-27T15:38:30.8150247Z Running tests... 2022-09-27T15:38:30.8150835Z ---------------------------------------------------------------------- 2022-09-27T15:38:31.0625409Z test_no_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28016 2022-09-27T15:38:31.0652144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28017 2022-09-27T15:38:31.0680252Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28018 2022-09-27T15:38:31.0708991Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28019 2022-09-27T15:38:32.1848655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:32.1849300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:32.2051998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:32.2052688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:32.2220175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:32.2221070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:32.2546694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:32.2547673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:32.3138148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:32.3138825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:32.3492041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:32.3492647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:32.3518846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrmn3uq6 2022-09-27T15:38:32.3519990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrmn3uq6/_remote_module_non_scriptable.py 2022-09-27T15:38:32.3545610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:32.3546270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:32.3813757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifboxyc3 2022-09-27T15:38:32.3814606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifboxyc3/_remote_module_non_scriptable.py 2022-09-27T15:38:32.3906474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:32.3907109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:32.4755298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbehrhhl 2022-09-27T15:38:32.4756121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbehrhhl/_remote_module_non_scriptable.py 2022-09-27T15:38:32.5162969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprfbbnfyf 2022-09-27T15:38:32.5163890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprfbbnfyf/_remote_module_non_scriptable.py 2022-09-27T15:38:32.6097411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:32.6386821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:32.7336306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:38:32.7744137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:33.4768301Z ok (2.661s) 2022-09-27T15:38:33.4768561Z 2022-09-27T15:38:33.4769088Z ---------------------------------------------------------------------- 2022-09-27T15:38:33.4769438Z Ran 1 test in 2.662s 2022-09-27T15:38:33.4769542Z 2022-09-27T15:38:33.4769605Z OK 2022-09-27T15:38:33.4769714Z 2022-09-27T15:38:33.4769806Z Generating XML reports... 2022-09-27T15:38:33.4804303Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153830.xml 2022-09-27T15:38:34.8545020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:34.8545408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:34.8554464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:34.8554932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:35.0145346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpteqzh4du 2022-09-27T15:38:35.0146275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpteqzh4du/_remote_module_non_scriptable.py 2022-09-27T15:38:35.2774392Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:35.2784489Z 2022-09-27T15:38:35.2784701Z Running tests... 2022-09-27T15:38:35.2785178Z ---------------------------------------------------------------------- 2022-09-27T15:38:35.5262903Z test_record_function_jit_end_callbacks_with_fork (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28243 2022-09-27T15:38:35.5290702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28244 2022-09-27T15:38:35.5318147Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28245 2022-09-27T15:38:35.5346987Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28246 2022-09-27T15:38:36.6161325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:36.6161950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:36.6515478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:36.6516134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:36.6539354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:36.6539952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:36.6584684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:36.6585373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:36.6609342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:36.6609843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:36.6928435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:36.6929104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:36.6969575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:36.6970208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:36.6985765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:36.6986385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:36.7778891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbidv7i7 2022-09-27T15:38:36.7779702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbidv7i7/_remote_module_non_scriptable.py 2022-09-27T15:38:36.8184861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8r_v57wj 2022-09-27T15:38:36.8185872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8r_v57wj/_remote_module_non_scriptable.py 2022-09-27T15:38:36.8236634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19rlleyy 2022-09-27T15:38:36.8238178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19rlleyy/_remote_module_non_scriptable.py 2022-09-27T15:38:36.8239561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptzofd2_s 2022-09-27T15:38:36.8242661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptzofd2_s/_remote_module_non_scriptable.py 2022-09-27T15:38:37.0341013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:38:37.0738155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:37.0799809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:37.0814297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:38.4412114Z ok (3.162s) 2022-09-27T15:38:38.4412385Z 2022-09-27T15:38:38.4412901Z ---------------------------------------------------------------------- 2022-09-27T15:38:38.4413218Z Ran 1 test in 3.163s 2022-09-27T15:38:38.4413334Z 2022-09-27T15:38:38.4413396Z OK 2022-09-27T15:38:38.4413489Z 2022-09-27T15:38:38.4413587Z Generating XML reports... 2022-09-27T15:38:38.4450054Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153835.xml 2022-09-27T15:38:39.8165413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:39.8165789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:39.8173721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:39.8174217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:39.9746687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9mxxsha 2022-09-27T15:38:39.9747601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9mxxsha/_remote_module_non_scriptable.py 2022-09-27T15:38:40.2354749Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:40.2364434Z 2022-09-27T15:38:40.2364609Z Running tests... 2022-09-27T15:38:40.2365255Z ---------------------------------------------------------------------- 2022-09-27T15:38:40.4780065Z test_record_function_on_caller_rpc_async (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28298 2022-09-27T15:38:40.4806135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28299 2022-09-27T15:38:40.4833668Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28300 2022-09-27T15:38:40.4862515Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28301 2022-09-27T15:38:41.5542161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:41.5542781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:41.5742874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:41.5743483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:41.5841481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:41.5842181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:41.5908348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:41.5908922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:41.5936668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:41.5937193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:41.6139528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:41.6140091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:41.6241793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:41.6242447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:41.6331963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:41.6332879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:41.7188206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyphxo2zs 2022-09-27T15:38:41.7189223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyphxo2zs/_remote_module_non_scriptable.py 2022-09-27T15:38:41.7412266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwotjwiwp 2022-09-27T15:38:41.7413085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwotjwiwp/_remote_module_non_scriptable.py 2022-09-27T15:38:41.7531523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf163h30q 2022-09-27T15:38:41.7532254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf163h30q/_remote_module_non_scriptable.py 2022-09-27T15:38:41.7601211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadpzx7_3 2022-09-27T15:38:41.7602155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadpzx7_3/_remote_module_non_scriptable.py 2022-09-27T15:38:41.9792530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:42.0010695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:42.0174792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:42.0206335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:38:42.6919502Z ok (2.455s) 2022-09-27T15:38:42.6920783Z 2022-09-27T15:38:42.6921357Z ---------------------------------------------------------------------- 2022-09-27T15:38:42.6921837Z Ran 1 test in 2.455s 2022-09-27T15:38:42.6922033Z 2022-09-27T15:38:42.6922134Z OK 2022-09-27T15:38:42.6922284Z 2022-09-27T15:38:42.6922434Z Generating XML reports... 2022-09-27T15:38:42.6958254Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153840.xml 2022-09-27T15:38:44.0767871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:44.0768256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:44.0776719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:44.0777084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:44.2363750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjucamnxc 2022-09-27T15:38:44.2365179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjucamnxc/_remote_module_non_scriptable.py 2022-09-27T15:38:44.4961138Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:44.4971665Z 2022-09-27T15:38:44.4971917Z Running tests... 2022-09-27T15:38:44.4972391Z ---------------------------------------------------------------------- 2022-09-27T15:38:44.7469337Z test_remote_script_module (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28525 2022-09-27T15:38:44.7496856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28526 2022-09-27T15:38:44.7524594Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28527 2022-09-27T15:38:44.7553402Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28528 2022-09-27T15:38:45.8148665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:45.8149334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:45.8192321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:45.8193302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:45.8517732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:45.8518605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:45.8568663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:45.8569334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:45.8579070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:45.8579683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:45.8648357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:45.8648983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:45.8948180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:45.8948860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:45.9009368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:45.9010029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:45.9782526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhpz2qms 2022-09-27T15:38:45.9784056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhpz2qms/_remote_module_non_scriptable.py 2022-09-27T15:38:45.9821847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn16ybwqx 2022-09-27T15:38:45.9823513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn16ybwqx/_remote_module_non_scriptable.py 2022-09-27T15:38:46.0204332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9tdn_3k 2022-09-27T15:38:46.0205109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9tdn_3k/_remote_module_non_scriptable.py 2022-09-27T15:38:46.0258180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4ohb88h 2022-09-27T15:38:46.0259537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4ohb88h/_remote_module_non_scriptable.py 2022-09-27T15:38:46.2326627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:46.2351898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:46.2798268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:38:46.2806205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:47.0612611Z ok (2.564s) 2022-09-27T15:38:47.0612845Z 2022-09-27T15:38:47.0613270Z ---------------------------------------------------------------------- 2022-09-27T15:38:47.0613698Z Ran 1 test in 2.564s 2022-09-27T15:38:47.0613882Z 2022-09-27T15:38:47.0613982Z OK 2022-09-27T15:38:47.0614140Z 2022-09-27T15:38:47.0614294Z Generating XML reports... 2022-09-27T15:38:47.0650300Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153844.xml 2022-09-27T15:38:48.4324344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:48.4324795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:48.4334360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:48.4334770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:48.5956791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwd47v26 2022-09-27T15:38:48.5957663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwd47v26/_remote_module_non_scriptable.py 2022-09-27T15:38:48.8604103Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:48.8612986Z 2022-09-27T15:38:48.8613120Z Running tests... 2022-09-27T15:38:48.8613707Z ---------------------------------------------------------------------- 2022-09-27T15:38:49.1048922Z test_remote_script_throw (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28748 2022-09-27T15:38:49.1076370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28749 2022-09-27T15:38:49.1104277Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28750 2022-09-27T15:38:49.1133971Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28751 2022-09-27T15:38:50.2293786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:50.2294407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:50.2328273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:50.2328886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:50.2333868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:50.2334527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:50.2373015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:50.2373672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:50.2692701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:50.2693515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:50.2734855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:50.2735530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:50.2736334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:50.2736965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:50.2777511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:50.2778161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:50.3972163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputahqolu 2022-09-27T15:38:50.3973371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputahqolu/_remote_module_non_scriptable.py 2022-09-27T15:38:50.4017800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpextylgo4 2022-09-27T15:38:50.4018594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwmzu5u8 2022-09-27T15:38:50.4019513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpextylgo4/_remote_module_non_scriptable.py 2022-09-27T15:38:50.4020274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwmzu5u8/_remote_module_non_scriptable.py 2022-09-27T15:38:50.4051853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplyv5v6bl 2022-09-27T15:38:50.4053597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplyv5v6bl/_remote_module_non_scriptable.py 2022-09-27T15:38:50.6589602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:38:50.6590731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:50.6637552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:50.6647415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:51.4194032Z ok (2.558s) 2022-09-27T15:38:51.4194291Z 2022-09-27T15:38:51.4194928Z ---------------------------------------------------------------------- 2022-09-27T15:38:51.4195182Z Ran 1 test in 2.558s 2022-09-27T15:38:51.4195287Z 2022-09-27T15:38:51.4195347Z OK 2022-09-27T15:38:51.4195438Z 2022-09-27T15:38:51.4195538Z Generating XML reports... 2022-09-27T15:38:51.4230016Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153848.xml 2022-09-27T15:38:52.7875860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:52.7876572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:52.7885399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:52.7886005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:52.9473625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvm1lfg1o 2022-09-27T15:38:52.9474353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvm1lfg1o/_remote_module_non_scriptable.py 2022-09-27T15:38:53.2100899Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:53.2110828Z 2022-09-27T15:38:53.2111265Z Running tests... 2022-09-27T15:38:53.2111928Z ---------------------------------------------------------------------- 2022-09-27T15:38:53.4557771Z test_remote_script_udf (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28971 2022-09-27T15:38:53.4585275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28972 2022-09-27T15:38:53.4612300Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28973 2022-09-27T15:38:53.4641973Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28974 2022-09-27T15:38:54.5685191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:54.5685834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:54.5958124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:54.5958729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:54.6062203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:54.6062770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:54.6069181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:54.6069825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:54.6177690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:54.6178291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:54.6345682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:54.6346389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:54.6453320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:54.6453968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:54.6543582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:54.6544397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:54.7328143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fw0kn_k 2022-09-27T15:38:54.7328912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fw0kn_k/_remote_module_non_scriptable.py 2022-09-27T15:38:54.7608638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnupidk3g 2022-09-27T15:38:54.7609426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnupidk3g/_remote_module_non_scriptable.py 2022-09-27T15:38:54.7717912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwp_7dm6 2022-09-27T15:38:54.7718899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwp_7dm6/_remote_module_non_scriptable.py 2022-09-27T15:38:54.7800544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9x54yzo 2022-09-27T15:38:54.7801577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9x54yzo/_remote_module_non_scriptable.py 2022-09-27T15:38:54.9895080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:55.0180684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:55.0306841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:55.0385546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:38:55.7699964Z ok (2.559s) 2022-09-27T15:38:55.7700209Z 2022-09-27T15:38:55.7700569Z ---------------------------------------------------------------------- 2022-09-27T15:38:55.7700825Z Ran 1 test in 2.559s 2022-09-27T15:38:55.7700943Z 2022-09-27T15:38:55.7700991Z OK 2022-09-27T15:38:55.7701083Z 2022-09-27T15:38:55.7701196Z Generating XML reports... 2022-09-27T15:38:55.7737290Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153853.xml 2022-09-27T15:38:57.1299769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:57.1300412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:57.1308521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:57.1309132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:57.2844448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfooum7o 2022-09-27T15:38:57.2845463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfooum7o/_remote_module_non_scriptable.py 2022-09-27T15:38:57.5444529Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:38:57.5454174Z 2022-09-27T15:38:57.5454473Z Running tests... 2022-09-27T15:38:57.5455119Z ---------------------------------------------------------------------- 2022-09-27T15:38:57.7889687Z test_return_local_script_class_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29210 2022-09-27T15:38:57.7916918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29211 2022-09-27T15:38:57.7944239Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29212 2022-09-27T15:38:57.7973168Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29213 2022-09-27T15:38:58.9557565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:58.9558188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:58.9729310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:58.9729987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:58.9913265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:58.9913802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:59.0096038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:59.0096651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:59.0130792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:59.0131388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:59.0193344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:38:59.0193866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:38:59.0510403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:59.0511007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:59.0573489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:38:59.0574146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:38:59.1197934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptn6bmsq 2022-09-27T15:38:59.1198732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptn6bmsq/_remote_module_non_scriptable.py 2022-09-27T15:38:59.1364525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3mngz7_9 2022-09-27T15:38:59.1365493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3mngz7_9/_remote_module_non_scriptable.py 2022-09-27T15:38:59.1782425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6nbswkk 2022-09-27T15:38:59.1784320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6nbswkk/_remote_module_non_scriptable.py 2022-09-27T15:38:59.2025145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdsj0yaxd 2022-09-27T15:38:59.2026722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdsj0yaxd/_remote_module_non_scriptable.py 2022-09-27T15:38:59.3842033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:38:59.3981533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:38:59.4363878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:38:59.4621780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:00.2033552Z ok (2.658s) 2022-09-27T15:39:00.2033806Z 2022-09-27T15:39:00.2034271Z ---------------------------------------------------------------------- 2022-09-27T15:39:00.2034848Z Ran 1 test in 2.658s 2022-09-27T15:39:00.2035032Z 2022-09-27T15:39:00.2035095Z OK 2022-09-27T15:39:00.2035188Z 2022-09-27T15:39:00.2037039Z Generating XML reports... 2022-09-27T15:39:00.2070632Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153857.xml 2022-09-27T15:39:01.5918992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:01.5919371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:01.5928337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:01.5928905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:01.7567897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthh4w4k5 2022-09-27T15:39:01.7568746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthh4w4k5/_remote_module_non_scriptable.py 2022-09-27T15:39:02.0224033Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:02.0233517Z 2022-09-27T15:39:02.0233641Z Running tests... 2022-09-27T15:39:02.0234043Z ---------------------------------------------------------------------- 2022-09-27T15:39:02.2728201Z test_return_local_script_module_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29437 2022-09-27T15:39:02.2755411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29438 2022-09-27T15:39:02.2782865Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29439 2022-09-27T15:39:02.2812227Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29440 2022-09-27T15:39:03.4172451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:03.4173080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:03.4397446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:03.4398108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:03.4464299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:03.4464971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:03.4549885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:03.4550486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:03.4551189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:03.4551791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:03.4779744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:03.4780273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:03.4844439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:03.4845118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:03.4923788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:03.4924453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:03.5821150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10c9jkhv 2022-09-27T15:39:03.5822290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10c9jkhv/_remote_module_non_scriptable.py 2022-09-27T15:39:03.6046874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpghjfbr4l 2022-09-27T15:39:03.6047683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpghjfbr4l/_remote_module_non_scriptable.py 2022-09-27T15:39:03.6108108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehbuqo07 2022-09-27T15:39:03.6108862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehbuqo07/_remote_module_non_scriptable.py 2022-09-27T15:39:03.6182855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5w0xneq 2022-09-27T15:39:03.6184412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5w0xneq/_remote_module_non_scriptable.py 2022-09-27T15:39:03.8395614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:39:03.8583709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:03.8643081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:39:03.8750510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:39:04.6873097Z ok (2.663s) 2022-09-27T15:39:04.6873353Z 2022-09-27T15:39:04.6873858Z ---------------------------------------------------------------------- 2022-09-27T15:39:04.6874247Z Ran 1 test in 2.664s 2022-09-27T15:39:04.6874366Z 2022-09-27T15:39:04.6874425Z OK 2022-09-27T15:39:04.6874669Z 2022-09-27T15:39:04.6874813Z Generating XML reports... 2022-09-27T15:39:04.6909493Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153902.xml 2022-09-27T15:39:06.0418921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:06.0419304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:06.0426938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:06.0427475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:06.1963464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1kvmx75o 2022-09-27T15:39:06.1964379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1kvmx75o/_remote_module_non_scriptable.py 2022-09-27T15:39:06.4566959Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:06.4576133Z 2022-09-27T15:39:06.4576390Z Running tests... 2022-09-27T15:39:06.4576784Z ---------------------------------------------------------------------- 2022-09-27T15:39:06.7016866Z test_rpc_async_jit_profiled (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29664 2022-09-27T15:39:06.7043446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29665 2022-09-27T15:39:06.7070372Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29666 2022-09-27T15:39:06.7100834Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29667 2022-09-27T15:39:07.8076859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:07.8077443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:07.8094185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:07.8094792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:07.8406218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:07.8406879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:07.8407684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:07.8408291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:07.8471797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:07.8472309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:07.8495830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:07.8496370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:07.8812729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:07.8813423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:07.8814349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:07.8814938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:07.9753706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5_mut76 2022-09-27T15:39:07.9754394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5_mut76/_remote_module_non_scriptable.py 2022-09-27T15:39:07.9767710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphdgmvea1 2022-09-27T15:39:07.9768965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphdgmvea1/_remote_module_non_scriptable.py 2022-09-27T15:39:08.0096187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3p5qvwbv 2022-09-27T15:39:08.0096973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfbvdahj 2022-09-27T15:39:08.0097729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3p5qvwbv/_remote_module_non_scriptable.py 2022-09-27T15:39:08.0098476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfbvdahj/_remote_module_non_scriptable.py 2022-09-27T15:39:08.2324540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:39:08.2331187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:39:08.2675138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:08.2684575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:39:09.0158087Z ok (2.558s) 2022-09-27T15:39:09.0158303Z 2022-09-27T15:39:09.0158653Z ---------------------------------------------------------------------- 2022-09-27T15:39:09.0158934Z Ran 1 test in 2.558s 2022-09-27T15:39:09.0159082Z 2022-09-27T15:39:09.0159151Z OK 2022-09-27T15:39:09.0159243Z 2022-09-27T15:39:09.0159348Z Generating XML reports... 2022-09-27T15:39:09.0194361Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153906.xml 2022-09-27T15:39:10.4003658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:10.4004336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:10.4012796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:10.4013432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:10.5616889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphy15mw07 2022-09-27T15:39:10.5618047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphy15mw07/_remote_module_non_scriptable.py 2022-09-27T15:39:10.8229204Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:10.8239416Z 2022-09-27T15:39:10.8239796Z Running tests... 2022-09-27T15:39:10.8240409Z ---------------------------------------------------------------------- 2022-09-27T15:39:11.0653000Z test_rpc_torchscript_record_function (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29891 2022-09-27T15:39:11.0680376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29892 2022-09-27T15:39:11.0707917Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29893 2022-09-27T15:39:11.0736974Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29894 2022-09-27T15:39:12.1444990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:12.1445764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:12.1604293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:12.1604905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:12.1626354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:12.1627030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:12.1780202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:12.1780922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:12.1826847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:12.1827413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:12.1996407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:12.1997170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:12.2016624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:12.2017280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:12.2157223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:12.2157853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:12.3106020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7bsqyrc 2022-09-27T15:39:12.3106850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7bsqyrc/_remote_module_non_scriptable.py 2022-09-27T15:39:12.3268872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufe8a7pa 2022-09-27T15:39:12.3269714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufe8a7pa/_remote_module_non_scriptable.py 2022-09-27T15:39:12.3283120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0t_ac8rc 2022-09-27T15:39:12.3285870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0t_ac8rc/_remote_module_non_scriptable.py 2022-09-27T15:39:12.3429154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7mq1ga8 2022-09-27T15:39:12.3430424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7mq1ga8/_remote_module_non_scriptable.py 2022-09-27T15:39:12.5666605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:39:12.5810520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:12.5841944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:39:12.5994895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:39:13.3797693Z ok (2.555s) 2022-09-27T15:39:13.3797935Z 2022-09-27T15:39:13.3798334Z ---------------------------------------------------------------------- 2022-09-27T15:39:13.3798609Z Ran 1 test in 2.556s 2022-09-27T15:39:13.3798723Z 2022-09-27T15:39:13.3798785Z OK 2022-09-27T15:39:13.3798876Z 2022-09-27T15:39:13.3798983Z Generating XML reports... 2022-09-27T15:39:13.3833911Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153910.xml 2022-09-27T15:39:14.7502713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:14.7503114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:14.7510974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:14.7511475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:14.9065523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4329c49 2022-09-27T15:39:14.9066510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4329c49/_remote_module_non_scriptable.py 2022-09-27T15:39:15.1660361Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:15.1669953Z 2022-09-27T15:39:15.1670078Z Running tests... 2022-09-27T15:39:15.1670489Z ---------------------------------------------------------------------- 2022-09-27T15:39:15.4099926Z test_rref_as_arg_and_return (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30118 2022-09-27T15:39:15.4126915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30119 2022-09-27T15:39:15.4154073Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30120 2022-09-27T15:39:15.4182829Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30121 2022-09-27T15:39:16.5289801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:16.5290440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:16.5294596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:16.5295248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:16.5323819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:16.5324460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:16.5609568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:16.5610225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:16.5665040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:16.5665772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:16.5681061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:16.5681634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:16.5702760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:16.5703386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:16.5958412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:16.5959021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:16.6932718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0dzfwb2 2022-09-27T15:39:16.6933499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0dzfwb2/_remote_module_non_scriptable.py 2022-09-27T15:39:16.6950137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_sqx41zn 2022-09-27T15:39:16.6952123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_sqx41zn/_remote_module_non_scriptable.py 2022-09-27T15:39:16.6965373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrkpamys 2022-09-27T15:39:16.6967263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrkpamys/_remote_module_non_scriptable.py 2022-09-27T15:39:16.7230582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33yzgaa1 2022-09-27T15:39:16.7231500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33yzgaa1/_remote_module_non_scriptable.py 2022-09-27T15:39:16.9506926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:39:16.9559425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:39:16.9643309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:16.9773084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:39:17.7246877Z ok (2.557s) 2022-09-27T15:39:17.7247151Z 2022-09-27T15:39:17.7247604Z ---------------------------------------------------------------------- 2022-09-27T15:39:17.7247861Z Ran 1 test in 2.557s 2022-09-27T15:39:17.7247979Z 2022-09-27T15:39:17.7248039Z OK 2022-09-27T15:39:17.7248130Z 2022-09-27T15:39:17.7248223Z Generating XML reports... 2022-09-27T15:39:17.7284567Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153915.xml 2022-09-27T15:39:19.1071574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:19.1072237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:19.1081056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:19.1081656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:19.2644278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xbccehp 2022-09-27T15:39:19.2645183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xbccehp/_remote_module_non_scriptable.py 2022-09-27T15:39:19.5251185Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:19.5261022Z 2022-09-27T15:39:19.5261363Z Running tests... 2022-09-27T15:39:19.5261971Z ---------------------------------------------------------------------- 2022-09-27T15:39:19.7632523Z test_rref_is_owner (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30341 2022-09-27T15:39:19.7660262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30342 2022-09-27T15:39:19.7688423Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30343 2022-09-27T15:39:19.7716810Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30344 2022-09-27T15:39:20.8976692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:20.8977319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:20.9215670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:20.9216355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:20.9221711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:20.9222353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:20.9269221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:20.9269873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:20.9346131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:20.9346660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:20.9594291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:20.9595504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:20.9600085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:20.9600977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:20.9647492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:20.9648102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:21.0620350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy81xqjbd 2022-09-27T15:39:21.0621133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy81xqjbd/_remote_module_non_scriptable.py 2022-09-27T15:39:21.0861823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23ioc_3h 2022-09-27T15:39:21.0862705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23ioc_3h/_remote_module_non_scriptable.py 2022-09-27T15:39:21.0864874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1z0_htlh 2022-09-27T15:39:21.0866998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1z0_htlh/_remote_module_non_scriptable.py 2022-09-27T15:39:21.0902708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpljy7pla8 2022-09-27T15:39:21.0904572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpljy7pla8/_remote_module_non_scriptable.py 2022-09-27T15:39:21.3201087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:21.3419947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:39:21.3420540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:39:21.3448406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:39:22.0776024Z ok (2.551s) 2022-09-27T15:39:22.0776242Z 2022-09-27T15:39:22.0776687Z ---------------------------------------------------------------------- 2022-09-27T15:39:22.0777095Z Ran 1 test in 2.551s 2022-09-27T15:39:22.0777296Z 2022-09-27T15:39:22.0777402Z OK 2022-09-27T15:39:22.0777548Z 2022-09-27T15:39:22.0777670Z Generating XML reports... 2022-09-27T15:39:22.0813650Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153919.xml 2022-09-27T15:39:23.4430852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:23.4431231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:23.4439490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:23.4439863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:23.5999089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8x5ez1u0 2022-09-27T15:39:23.5999962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8x5ez1u0/_remote_module_non_scriptable.py 2022-09-27T15:39:23.8603062Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:23.8612610Z 2022-09-27T15:39:23.8612698Z Running tests... 2022-09-27T15:39:23.8613152Z ---------------------------------------------------------------------- 2022-09-27T15:39:24.1031963Z test_rref_jit_pickle_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30564 2022-09-27T15:39:24.1058696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30565 2022-09-27T15:39:24.1086340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30566 2022-09-27T15:39:24.1117155Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30567 2022-09-27T15:39:25.1855309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:25.1856263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:25.1885718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:25.1886380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:25.2239995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:25.2240795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:25.2273872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:25.2274762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:25.2292968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:25.2293616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:25.2294399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:25.2295009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:25.2678312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:25.2678879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:25.2681782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:25.2682335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:25.3523102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgyzrmt2g 2022-09-27T15:39:25.3523897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgyzrmt2g/_remote_module_non_scriptable.py 2022-09-27T15:39:25.3552063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95z30arl 2022-09-27T15:39:25.3553975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95z30arl/_remote_module_non_scriptable.py 2022-09-27T15:39:25.3954207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuygvgffn 2022-09-27T15:39:25.3955029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphi2r5keg 2022-09-27T15:39:25.3955694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuygvgffn/_remote_module_non_scriptable.py 2022-09-27T15:39:25.3956365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphi2r5keg/_remote_module_non_scriptable.py 2022-09-27T15:39:25.6150214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:39:25.6163928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:25.6549883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:39:25.6568164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:39:26.4174587Z ok (2.556s) 2022-09-27T15:39:26.4174850Z 2022-09-27T15:39:26.4175360Z ---------------------------------------------------------------------- 2022-09-27T15:39:26.4175675Z Ran 1 test in 2.556s 2022-09-27T15:39:26.4175791Z 2022-09-27T15:39:26.4175853Z OK 2022-09-27T15:39:26.4175943Z 2022-09-27T15:39:26.4176024Z Generating XML reports... 2022-09-27T15:39:26.4211865Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153923.xml 2022-09-27T15:39:27.7917549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:27.7918217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:27.7926918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:27.7927561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:27.9487635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8sntv7k 2022-09-27T15:39:27.9488693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8sntv7k/_remote_module_non_scriptable.py 2022-09-27T15:39:28.2113602Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:28.2122599Z 2022-09-27T15:39:28.2122729Z Running tests... 2022-09-27T15:39:28.2123459Z ---------------------------------------------------------------------- 2022-09-27T15:39:28.4568511Z test_rref_list_mutate (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30787 2022-09-27T15:39:28.4595097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30788 2022-09-27T15:39:28.4622528Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30789 2022-09-27T15:39:28.4651665Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30790 2022-09-27T15:39:29.5340004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:29.5340649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:29.5552132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:29.5552830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:29.5703185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:29.5703714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:29.5928049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:29.5981069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:29.5982605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:29.5983228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:29.5984028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:29.5984645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:29.6358779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:29.6359373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:29.6360097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:29.6360673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:29.6992384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddvevoya 2022-09-27T15:39:29.6993337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddvevoya/_remote_module_non_scriptable.py 2022-09-27T15:39:29.7201832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5pnlblqu 2022-09-27T15:39:29.7202630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5pnlblqu/_remote_module_non_scriptable.py 2022-09-27T15:39:29.7629860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7aan9tq3 2022-09-27T15:39:29.7630835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9y2xx_va 2022-09-27T15:39:29.7631651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7aan9tq3/_remote_module_non_scriptable.py 2022-09-27T15:39:29.7632497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9y2xx_va/_remote_module_non_scriptable.py 2022-09-27T15:39:29.9590146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:39:29.9778684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:39:30.0185232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:39:30.0186458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:30.7709403Z ok (2.558s) 2022-09-27T15:39:30.7709664Z 2022-09-27T15:39:30.7710176Z ---------------------------------------------------------------------- 2022-09-27T15:39:30.7710468Z Ran 1 test in 2.559s 2022-09-27T15:39:30.7710599Z 2022-09-27T15:39:30.7710662Z OK 2022-09-27T15:39:30.7710756Z 2022-09-27T15:39:30.7710853Z Generating XML reports... 2022-09-27T15:39:30.7747408Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153928.xml 2022-09-27T15:39:32.1970780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:32.1971167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:32.1980945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:32.1981340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:32.3625756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbn874kuh 2022-09-27T15:39:32.3626955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbn874kuh/_remote_module_non_scriptable.py 2022-09-27T15:39:32.6292992Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:32.6302929Z 2022-09-27T15:39:32.6303198Z Running tests... 2022-09-27T15:39:32.6303606Z ---------------------------------------------------------------------- 2022-09-27T15:39:32.8809370Z test_rref_local_value (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31010 2022-09-27T15:39:32.8837790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31011 2022-09-27T15:39:32.8866880Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31012 2022-09-27T15:39:32.8897005Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31013 2022-09-27T15:39:33.9916919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:33.9917579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:34.0290342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:34.0291013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:34.0484246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:34.0484922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:34.0498662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:34.0499287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:34.0789448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:34.0790091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:34.0876982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:34.0877693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:34.0890797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:34.0891355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:34.1167143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:34.1167766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:34.1577929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5yjdxb51 2022-09-27T15:39:34.1578678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5yjdxb51/_remote_module_non_scriptable.py 2022-09-27T15:39:34.2159743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7k01wo_w 2022-09-27T15:39:34.2160526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7k01wo_w/_remote_module_non_scriptable.py 2022-09-27T15:39:34.2161553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2kq3fg3 2022-09-27T15:39:34.2164266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2kq3fg3/_remote_module_non_scriptable.py 2022-09-27T15:39:34.2438765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmhdjq7a 2022-09-27T15:39:34.2440081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmhdjq7a/_remote_module_non_scriptable.py 2022-09-27T15:39:34.4388121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:34.4765206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:39:34.4799784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:39:34.5044925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:39:35.2958808Z ok (2.665s) 2022-09-27T15:39:35.2959061Z 2022-09-27T15:39:35.2959533Z ---------------------------------------------------------------------- 2022-09-27T15:39:35.2960000Z Ran 1 test in 2.665s 2022-09-27T15:39:35.2960209Z 2022-09-27T15:39:35.2960323Z OK 2022-09-27T15:39:35.2960487Z 2022-09-27T15:39:35.2960636Z Generating XML reports... 2022-09-27T15:39:35.2994864Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153932.xml 2022-09-27T15:39:36.6641990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:36.6642368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:36.6649917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:36.6650438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:36.8224073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkwr6at9 2022-09-27T15:39:36.8225098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkwr6at9/_remote_module_non_scriptable.py 2022-09-27T15:39:37.0872764Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:37.0883243Z 2022-09-27T15:39:37.0883337Z Running tests... 2022-09-27T15:39:37.0884493Z ---------------------------------------------------------------------- 2022-09-27T15:39:37.3387736Z test_rref_python_annotation (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31233 2022-09-27T15:39:37.3416043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31234 2022-09-27T15:39:37.3444503Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31235 2022-09-27T15:39:37.3474196Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31236 2022-09-27T15:39:38.4726944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:38.4727529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:38.4784675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:38.4785270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:38.4832795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:38.4833445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:38.4939930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:38.4940449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:38.5123139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:38.5123799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:38.5182335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:38.5183002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:38.5225826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:38.5226466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:38.5325036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:38.5325669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:38.6396726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpae4k3ol_ 2022-09-27T15:39:38.6397774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpae4k3ol_/_remote_module_non_scriptable.py 2022-09-27T15:39:38.6449754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4jretew 2022-09-27T15:39:38.6451366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4jretew/_remote_module_non_scriptable.py 2022-09-27T15:39:38.6488086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyg9hgzgr 2022-09-27T15:39:38.6489897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyg9hgzgr/_remote_module_non_scriptable.py 2022-09-27T15:39:38.6591089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiln9wgco 2022-09-27T15:39:38.6592493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiln9wgco/_remote_module_non_scriptable.py 2022-09-27T15:39:38.8996759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:39:38.9059288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:38.9081478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:39:38.9206993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:39:39.6533092Z ok (2.565s) 2022-09-27T15:39:39.6533286Z 2022-09-27T15:39:39.6533583Z ---------------------------------------------------------------------- 2022-09-27T15:39:39.6533850Z Ran 1 test in 2.565s 2022-09-27T15:39:39.6533967Z 2022-09-27T15:39:39.6534016Z OK 2022-09-27T15:39:39.6534108Z 2022-09-27T15:39:39.6534202Z Generating XML reports... 2022-09-27T15:39:39.6569740Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153937.xml 2022-09-27T15:39:41.0201256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:41.0201945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:41.0210911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:41.0211259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:41.1830487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt33inexi 2022-09-27T15:39:41.1831754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt33inexi/_remote_module_non_scriptable.py 2022-09-27T15:39:41.4500438Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:41.4510426Z 2022-09-27T15:39:41.4510511Z Running tests... 2022-09-27T15:39:41.4511478Z ---------------------------------------------------------------------- 2022-09-27T15:39:41.7014405Z test_some_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31456 2022-09-27T15:39:41.7041888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31457 2022-09-27T15:39:41.7069067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31458 2022-09-27T15:39:41.7098973Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31459 2022-09-27T15:39:42.8048473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:42.8049088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:42.8055313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:42.8056054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:42.8183358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:42.8184052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:42.8429353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:42.8429998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:42.8430692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:42.8431196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:42.8440184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:42.8440766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:42.8565824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:42.8566478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:42.8819854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:42.8820514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:42.9707807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79r2x84m 2022-09-27T15:39:42.9709737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79r2x84m/_remote_module_non_scriptable.py 2022-09-27T15:39:42.9710419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppz8jcbu8 2022-09-27T15:39:42.9713220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppz8jcbu8/_remote_module_non_scriptable.py 2022-09-27T15:39:42.9824580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1a4ej0sb 2022-09-27T15:39:42.9825581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1a4ej0sb/_remote_module_non_scriptable.py 2022-09-27T15:39:43.0072978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr87h44r0 2022-09-27T15:39:43.0073727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr87h44r0/_remote_module_non_scriptable.py 2022-09-27T15:39:43.2255394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:43.2281051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:39:43.2395136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:39:43.2635346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:39:44.0159315Z ok (2.565s) 2022-09-27T15:39:44.0159519Z 2022-09-27T15:39:44.0159925Z ---------------------------------------------------------------------- 2022-09-27T15:39:44.0160181Z Ran 1 test in 2.565s 2022-09-27T15:39:44.0160305Z 2022-09-27T15:39:44.0160382Z OK 2022-09-27T15:39:44.0160519Z 2022-09-27T15:39:44.0160608Z Generating XML reports... 2022-09-27T15:39:44.0196363Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153941.xml 2022-09-27T15:39:45.3876079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:45.3876447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:45.3884151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:45.3884516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:45.5430991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgw8pscn0 2022-09-27T15:39:45.5432397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgw8pscn0/_remote_module_non_scriptable.py 2022-09-27T15:39:45.8042118Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:45.8051516Z 2022-09-27T15:39:45.8051668Z Running tests... 2022-09-27T15:39:45.8052090Z ---------------------------------------------------------------------- 2022-09-27T15:39:46.0482335Z test_torchscript_function (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31683 2022-09-27T15:39:46.0508836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31684 2022-09-27T15:39:46.0537438Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31685 2022-09-27T15:39:46.0566412Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31686 2022-09-27T15:39:47.2025640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:47.2026269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:47.2399132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:47.2399792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:47.2517882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:47.2518579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:47.2800285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:47.2800959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:47.2903423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:47.2904052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:47.2979352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:47.2979982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:47.3192084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:47.3192717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:47.3349201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:47.3349873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:47.3685858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl5bp7ddn 2022-09-27T15:39:47.3686680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl5bp7ddn/_remote_module_non_scriptable.py 2022-09-27T15:39:47.4175794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwividrdh 2022-09-27T15:39:47.4176523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwividrdh/_remote_module_non_scriptable.py 2022-09-27T15:39:47.4457914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe56noe5z 2022-09-27T15:39:47.4458793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe56noe5z/_remote_module_non_scriptable.py 2022-09-27T15:39:47.4612190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwch1ay7h 2022-09-27T15:39:47.4613013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwch1ay7h/_remote_module_non_scriptable.py 2022-09-27T15:39:47.6273328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:39:47.6768360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:47.7082003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:39:47.7235212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:39:48.4626845Z ok (2.657s) 2022-09-27T15:39:48.4627218Z 2022-09-27T15:39:48.4627756Z ---------------------------------------------------------------------- 2022-09-27T15:39:48.4628222Z Ran 1 test in 2.657s 2022-09-27T15:39:48.4628411Z 2022-09-27T15:39:48.4628460Z OK 2022-09-27T15:39:48.4628559Z 2022-09-27T15:39:48.4628656Z Generating XML reports... 2022-09-27T15:39:48.4664089Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153945.xml 2022-09-27T15:39:49.8656336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:49.8656811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:49.8665192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:49.8665589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:50.0248143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwjm5calc 2022-09-27T15:39:50.0248721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwjm5calc/_remote_module_non_scriptable.py 2022-09-27T15:39:50.2873138Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:50.2883133Z 2022-09-27T15:39:50.2883230Z Running tests... 2022-09-27T15:39:50.2883817Z ---------------------------------------------------------------------- 2022-09-27T15:39:50.5313827Z test_torchscript_function_exception (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31906 2022-09-27T15:39:50.5340029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31907 2022-09-27T15:39:50.5367896Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31908 2022-09-27T15:39:50.5397237Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31909 2022-09-27T15:39:51.6582459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:51.6583042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:51.6779029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:51.6779669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:51.6783407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:51.6783954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:51.6831236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:51.6831851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:51.6958279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:51.6958902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:51.7169253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:51.7169902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:51.7170616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:51.7171182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:51.7211750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:51.7212409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:51.8215819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3tav8__d 2022-09-27T15:39:51.8216617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3tav8__d/_remote_module_non_scriptable.py 2022-09-27T15:39:51.8433099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_momjhq 2022-09-27T15:39:51.8435799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_momjhq/_remote_module_non_scriptable.py 2022-09-27T15:39:51.8461300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw51z63zb 2022-09-27T15:39:51.8463791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw51z63zb/_remote_module_non_scriptable.py 2022-09-27T15:39:51.8533351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc32hw5c0 2022-09-27T15:39:51.8535434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc32hw5c0/_remote_module_non_scriptable.py 2022-09-27T15:39:52.0824690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:39:52.1031636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:52.1053733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:39:52.1202899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:39:52.7453846Z ok (2.457s) 2022-09-27T15:39:52.7454035Z 2022-09-27T15:39:52.7454328Z ---------------------------------------------------------------------- 2022-09-27T15:39:52.7454589Z Ran 1 test in 2.457s 2022-09-27T15:39:52.7454710Z 2022-09-27T15:39:52.7454771Z OK 2022-09-27T15:39:52.7454863Z 2022-09-27T15:39:52.7455202Z Generating XML reports... 2022-09-27T15:39:52.7490219Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153950.xml 2022-09-27T15:39:54.1379530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:54.1379896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:54.1387778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:54.1388138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:54.2997740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqpocd_f 2022-09-27T15:39:54.2998681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqpocd_f/_remote_module_non_scriptable.py 2022-09-27T15:39:54.5638736Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:54.5647745Z 2022-09-27T15:39:54.5648036Z Running tests... 2022-09-27T15:39:54.5648710Z ---------------------------------------------------------------------- 2022-09-27T15:39:54.8103603Z test_torchscript_functions_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32129 2022-09-27T15:39:54.8131647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32130 2022-09-27T15:39:54.8160463Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32131 2022-09-27T15:39:54.8191024Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32132 2022-09-27T15:39:55.9315520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:55.9316201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:55.9374126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:55.9374762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:55.9462985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:55.9463678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:55.9553021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:55.9553506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:55.9698532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:55.9699075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:55.9773096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:55.9773647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:55.9846993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:55.9847667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:55.9937734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:55.9938410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:56.0972096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptaxhr_k7 2022-09-27T15:39:56.0973262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptaxhr_k7/_remote_module_non_scriptable.py 2022-09-27T15:39:56.1127709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyis2cqu 2022-09-27T15:39:56.1128578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyis2cqu/_remote_module_non_scriptable.py 2022-09-27T15:39:56.1162085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcvhc3enm 2022-09-27T15:39:56.1164142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcvhc3enm/_remote_module_non_scriptable.py 2022-09-27T15:39:56.1212298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzir9j1ja 2022-09-27T15:39:56.1214548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzir9j1ja/_remote_module_non_scriptable.py 2022-09-27T15:39:56.3532687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:39:56.3685086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:39:56.3732201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:39:56.3802894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:39:56.6137960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:39:56.6244893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:39:56.6340819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:39:56.6346259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:39:56.6347657Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:39:56.6348454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:39:56.6442771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:39:56.6443471Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:39:57.1249471Z ok (2.560s) 2022-09-27T15:39:57.1249684Z 2022-09-27T15:39:57.1250080Z ---------------------------------------------------------------------- 2022-09-27T15:39:57.1250400Z Ran 1 test in 2.560s 2022-09-27T15:39:57.1250520Z 2022-09-27T15:39:57.1250583Z OK 2022-09-27T15:39:57.1250679Z 2022-09-27T15:39:57.1250774Z Generating XML reports... 2022-09-27T15:39:57.1286909Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153954.xml 2022-09-27T15:39:58.5196964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:39:58.5197371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:39:58.5204969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:39:58.5205505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:39:58.6769876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qvqrump 2022-09-27T15:39:58.6770674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qvqrump/_remote_module_non_scriptable.py 2022-09-27T15:39:58.9396895Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:39:58.9406211Z 2022-09-27T15:39:58.9406471Z Running tests... 2022-09-27T15:39:58.9407062Z ---------------------------------------------------------------------- 2022-09-27T15:39:59.1952448Z test_unexepected_kwarg_is_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32364 2022-09-27T15:39:59.1981595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32365 2022-09-27T15:39:59.2010893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32366 2022-09-27T15:39:59.2046593Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32367 2022-09-27T15:40:00.3058647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:00.3059336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:00.3175126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:00.3175832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:00.3284178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:00.3284741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:00.3441894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:00.3442556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:00.3536778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:00.3537458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:00.3554340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:00.3555202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:00.3660294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:00.3660920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:00.3901654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:00.3902389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:00.4722788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7fxdj_r 2022-09-27T15:40:00.4724585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7fxdj_r/_remote_module_non_scriptable.py 2022-09-27T15:40:00.4885379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9dxr18zq 2022-09-27T15:40:00.4887765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9dxr18zq/_remote_module_non_scriptable.py 2022-09-27T15:40:00.4925904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrkl5o5t 2022-09-27T15:40:00.4928223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrkl5o5t/_remote_module_non_scriptable.py 2022-09-27T15:40:00.5156685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzr237f7y 2022-09-27T15:40:00.5157543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzr237f7y/_remote_module_non_scriptable.py 2022-09-27T15:40:00.7488177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:00.7507343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:40:00.7618664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:00.7710382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:40:01.5105741Z ok (2.570s) 2022-09-27T15:40:01.5105917Z 2022-09-27T15:40:01.5106277Z ---------------------------------------------------------------------- 2022-09-27T15:40:01.5106533Z Ran 1 test in 2.570s 2022-09-27T15:40:01.5106647Z 2022-09-27T15:40:01.5106695Z OK 2022-09-27T15:40:01.5106786Z 2022-09-27T15:40:01.5106880Z Generating XML reports... 2022-09-27T15:40:01.5142747Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153958.xml 2022-09-27T15:40:02.9323968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:02.9324658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:02.9331948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:02.9332313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:03.0906794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdq8v8w6q 2022-09-27T15:40:03.0907493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdq8v8w6q/_remote_module_non_scriptable.py 2022-09-27T15:40:03.3538546Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:03.3547978Z 2022-09-27T15:40:03.3548118Z Running tests... 2022-09-27T15:40:03.3548714Z ---------------------------------------------------------------------- 2022-09-27T15:40:03.5965887Z test_user_rrefs_confirmed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32587 2022-09-27T15:40:03.5992768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32588 2022-09-27T15:40:03.6019816Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32589 2022-09-27T15:40:03.6049646Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32590 2022-09-27T15:40:04.7242867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:04.7243494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:04.7472090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:04.7472718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:04.7531681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:04.7532342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:04.7614428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:04.7615050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:04.7719828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:04.7720482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:04.7863290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:04.7863929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:04.7923876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:04.7924528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:04.8097025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:04.8097613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:04.8896185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6o2vto4 2022-09-27T15:40:04.8897325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6o2vto4/_remote_module_non_scriptable.py 2022-09-27T15:40:04.9137517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4dgobjeu 2022-09-27T15:40:04.9138494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4dgobjeu/_remote_module_non_scriptable.py 2022-09-27T15:40:04.9186461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8w_0vpm 2022-09-27T15:40:04.9188262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8w_0vpm/_remote_module_non_scriptable.py 2022-09-27T15:40:04.9359178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpor94cwg6 2022-09-27T15:40:04.9360034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpor94cwg6/_remote_module_non_scriptable.py 2022-09-27T15:40:05.1453125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:05.1705837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:05.1719837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:40:05.1927050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:40:05.9109188Z ok (2.556s) 2022-09-27T15:40:05.9109490Z 2022-09-27T15:40:05.9110025Z ---------------------------------------------------------------------- 2022-09-27T15:40:05.9110398Z Ran 1 test in 2.556s 2022-09-27T15:40:05.9110518Z 2022-09-27T15:40:05.9110579Z OK 2022-09-27T15:40:05.9110659Z 2022-09-27T15:40:05.9110754Z Generating XML reports... 2022-09-27T15:40:05.9146056Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927154003.xml 2022-09-27T15:40:07.2774011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:07.2774446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:07.2781636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:07.2781997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:07.4347850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46vs3j9w 2022-09-27T15:40:07.4348687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46vs3j9w/_remote_module_non_scriptable.py 2022-09-27T15:40:07.7007547Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:07.7017265Z 2022-09-27T15:40:07.7017376Z Running tests... 2022-09-27T15:40:07.7018050Z ---------------------------------------------------------------------- 2022-09-27T15:40:07.9489880Z test_user_rrefs_confirmed_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 342 2022-09-27T15:40:07.9516179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 343 2022-09-27T15:40:07.9543729Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 344 2022-09-27T15:40:07.9573324Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 345 2022-09-27T15:40:09.0824085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:09.0824766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:09.0900938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:09.0901692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:09.1173995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:09.1174673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:09.1200450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:09.1201125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:09.1280308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:09.1280883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:09.1300655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:09.1301246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:09.1554311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:09.1555205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:09.1676947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:09.1677582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:09.2486981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujoipa46 2022-09-27T15:40:09.2487748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujoipa46/_remote_module_non_scriptable.py 2022-09-27T15:40:09.2559567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu4l3j1dr 2022-09-27T15:40:09.2560387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu4l3j1dr/_remote_module_non_scriptable.py 2022-09-27T15:40:09.2831440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpva75ihgy 2022-09-27T15:40:09.2832193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpva75ihgy/_remote_module_non_scriptable.py 2022-09-27T15:40:09.2947855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmcowrp2o 2022-09-27T15:40:09.2948838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmcowrp2o/_remote_module_non_scriptable.py 2022-09-27T15:40:09.5077205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:09.5125820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:40:09.5518986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:40:09.5587564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:10.3634347Z ok (2.661s) 2022-09-27T15:40:10.3634648Z 2022-09-27T15:40:10.3634969Z ---------------------------------------------------------------------- 2022-09-27T15:40:10.3635219Z Ran 1 test in 2.662s 2022-09-27T15:40:10.3635360Z 2022-09-27T15:40:10.3635460Z OK 2022-09-27T15:40:10.3635553Z 2022-09-27T15:40:10.3635634Z Generating XML reports... 2022-09-27T15:40:10.3670599Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927154007.xml 2022-09-27T15:40:11.7262448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:11.7262854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:11.7271129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:11.7271567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:11.8871071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_6bffdg 2022-09-27T15:40:11.8872062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_6bffdg/_remote_module_non_scriptable.py 2022-09-27T15:40:12.1508209Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:12.1518321Z 2022-09-27T15:40:12.1518785Z Running tests... 2022-09-27T15:40:12.1519229Z ---------------------------------------------------------------------- 2022-09-27T15:40:12.3973495Z test_batch_updating_parameter_server (__main__.TensorPipeParameterServerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 565 2022-09-27T15:40:12.3999591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 566 2022-09-27T15:40:12.4027429Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 567 2022-09-27T15:40:12.4056353Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 568 2022-09-27T15:40:13.5658158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:13.5658754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:13.6020380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:13.6021059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:13.6200783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:13.6201461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:13.6208675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:13.6209324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:13.6317402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:13.6317936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:13.6581677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:13.6582426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:13.6594782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:13.6595397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:13.6694794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:13.6695475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:13.7306004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_myhg9a4 2022-09-27T15:40:13.7306889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_myhg9a4/_remote_module_non_scriptable.py 2022-09-27T15:40:13.7837186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9pw5q89 2022-09-27T15:40:13.7837999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9pw5q89/_remote_module_non_scriptable.py 2022-09-27T15:40:13.7853269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwmqiep20 2022-09-27T15:40:13.7855261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwmqiep20/_remote_module_non_scriptable.py 2022-09-27T15:40:13.7955495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxshgmw8c 2022-09-27T15:40:13.7957205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxshgmw8c/_remote_module_non_scriptable.py 2022-09-27T15:40:13.9880942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:40:14.0382997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:14.0426169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:14.0513438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:40:14.2616110Z 15:40:14 Start training 2022-09-27T15:40:14.2658973Z 15:40:14 worker3 processing one batch 2022-09-27T15:40:14.2665507Z 15:40:14 worker1 processing one batch 2022-09-27T15:40:14.2665975Z 15:40:14 worker2 processing one batch 2022-09-27T15:40:14.2670498Z 15:40:14 worker3 reporting grads 2022-09-27T15:40:14.2678249Z 15:40:14 worker1 reporting grads 2022-09-27T15:40:14.2678679Z 15:40:14 worker2 reporting grads 2022-09-27T15:40:14.2679691Z 15:40:14 PS got 0/3 updates 2022-09-27T15:40:14.2683663Z 15:40:14 PS got 1/3 updates 2022-09-27T15:40:14.2686566Z 15:40:14 PS got 2/3 updates 2022-09-27T15:40:14.2695299Z 15:40:14 PS updated model 2022-09-27T15:40:14.2697060Z 15:40:14 worker3 got updated model 2022-09-27T15:40:14.2697599Z 15:40:14 worker1 got updated model 2022-09-27T15:40:14.2698418Z 15:40:14 worker1 processing one batch 2022-09-27T15:40:14.2698873Z 15:40:14 worker3 processing one batch 2022-09-27T15:40:14.2702572Z 15:40:14 worker2 got updated model 2022-09-27T15:40:14.2703499Z 15:40:14 worker2 processing one batch 2022-09-27T15:40:14.2708169Z 15:40:14 worker2 reporting grads 2022-09-27T15:40:14.2708937Z 15:40:14 worker1 reporting grads 2022-09-27T15:40:14.2711609Z 15:40:14 worker3 reporting grads 2022-09-27T15:40:14.2712479Z 15:40:14 PS got 0/3 updates 2022-09-27T15:40:14.2714369Z 15:40:14 PS got 1/3 updates 2022-09-27T15:40:14.2718822Z 15:40:14 PS got 2/3 updates 2022-09-27T15:40:14.2726987Z 15:40:14 PS updated model 2022-09-27T15:40:14.2728869Z 15:40:14 worker2 got updated model 2022-09-27T15:40:14.2731619Z 15:40:14 worker2 processing one batch 2022-09-27T15:40:14.2733918Z 15:40:14 worker1 got updated model 2022-09-27T15:40:14.2734641Z 15:40:14 worker1 processing one batch 2022-09-27T15:40:14.2739522Z 15:40:14 worker1 reporting grads 2022-09-27T15:40:14.2739956Z 15:40:14 worker2 reporting grads 2022-09-27T15:40:14.2747325Z 15:40:14 worker3 got updated model 2022-09-27T15:40:14.2747731Z 15:40:14 worker3 processing one batch 2022-09-27T15:40:14.2749491Z 15:40:14 PS got 0/3 updates 2022-09-27T15:40:14.2749992Z 15:40:14 PS got 1/3 updates 2022-09-27T15:40:14.2753997Z 15:40:14 worker3 reporting grads 2022-09-27T15:40:14.2761460Z 15:40:14 PS got 2/3 updates 2022-09-27T15:40:14.2766941Z 15:40:14 PS updated model 2022-09-27T15:40:14.2769273Z 15:40:14 worker1 got updated model 2022-09-27T15:40:14.2769739Z 15:40:14 worker2 got updated model 2022-09-27T15:40:14.2771539Z 15:40:14 worker2 processing one batch 2022-09-27T15:40:14.2772256Z 15:40:14 worker1 processing one batch 2022-09-27T15:40:14.2773784Z 15:40:14 worker3 got updated model 2022-09-27T15:40:14.2774863Z 15:40:14 worker3 processing one batch 2022-09-27T15:40:14.2777819Z 15:40:14 worker2 reporting grads 2022-09-27T15:40:14.2778629Z 15:40:14 worker1 reporting grads 2022-09-27T15:40:14.2779391Z 15:40:14 worker3 reporting grads 2022-09-27T15:40:14.2785310Z 15:40:14 PS got 0/3 updates 2022-09-27T15:40:14.2790920Z 15:40:14 PS got 1/3 updates 2022-09-27T15:40:14.2792484Z 15:40:14 PS got 2/3 updates 2022-09-27T15:40:14.2800665Z 15:40:14 PS updated model 2022-09-27T15:40:14.2805335Z 15:40:14 worker3 got updated model 2022-09-27T15:40:14.2805900Z 15:40:14 worker2 got updated model 2022-09-27T15:40:14.2806297Z 15:40:14 worker1 got updated model 2022-09-27T15:40:14.2811447Z 15:40:14 Finish training 2022-09-27T15:40:14.2811890Z 15:40:14 Time spent training: 0.019680852001329185s 2022-09-27T15:40:14.7114266Z ok (2.559s) 2022-09-27T15:40:14.7114638Z 2022-09-27T15:40:14.7115106Z ---------------------------------------------------------------------- 2022-09-27T15:40:14.7115557Z Ran 1 test in 2.559s 2022-09-27T15:40:14.7115761Z 2022-09-27T15:40:14.7115834Z OK 2022-09-27T15:40:14.7115978Z 2022-09-27T15:40:14.7117755Z Generating XML reports... 2022-09-27T15:40:14.7152599Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeParameterServerTest-20220927154012.xml 2022-09-27T15:40:16.1016450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:16.1016834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:16.1024896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:16.1025620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:16.2644823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqqp_eg5 2022-09-27T15:40:16.2646052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqqp_eg5/_remote_module_non_scriptable.py 2022-09-27T15:40:16.5269377Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:16.5279597Z 2022-09-27T15:40:16.5279733Z Running tests... 2022-09-27T15:40:16.5280415Z ---------------------------------------------------------------------- 2022-09-27T15:40:16.7753580Z test_rl_rpc (__main__.TensorPipeReinforcementLearningRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 789 2022-09-27T15:40:16.7781526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 790 2022-09-27T15:40:16.7809416Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 791 2022-09-27T15:40:16.7839489Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 792 2022-09-27T15:40:17.9233502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:17.9234143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:17.9512279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:17.9512968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:17.9594291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:17.9595129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:17.9696692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:17.9697342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:17.9854441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:17.9855079Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:17.9875257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:17.9875895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:18.0066124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:18.0066777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:18.0216133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:18.0216732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:18.0868075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph452tb9c 2022-09-27T15:40:18.0868954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph452tb9c/_remote_module_non_scriptable.py 2022-09-27T15:40:18.1159386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphz8pm1t0 2022-09-27T15:40:18.1160145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphz8pm1t0/_remote_module_non_scriptable.py 2022-09-27T15:40:18.1349675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu308ch5n 2022-09-27T15:40:18.1350640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu308ch5n/_remote_module_non_scriptable.py 2022-09-27T15:40:18.1488124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmho7qheq 2022-09-27T15:40:18.1488868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmho7qheq/_remote_module_non_scriptable.py 2022-09-27T15:40:18.3476814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:18.3785909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:40:18.3979741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:40:18.4100652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:18.8763534Z Solved! Running reward is now 489.8051609821906! 2022-09-27T15:40:19.2898439Z ok (2.762s) 2022-09-27T15:40:19.2898646Z 2022-09-27T15:40:19.2899046Z ---------------------------------------------------------------------- 2022-09-27T15:40:19.2899464Z Ran 1 test in 2.762s 2022-09-27T15:40:19.2899679Z 2022-09-27T15:40:19.2899797Z OK 2022-09-27T15:40:19.2899935Z 2022-09-27T15:40:19.2900026Z Generating XML reports... 2022-09-27T15:40:19.2935167Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeReinforcementLearningRpcTest-20220927154016.xml 2022-09-27T15:40:20.6417853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:20.6418235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:20.6426431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:20.6426791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:20.7988907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnl1ke14f 2022-09-27T15:40:20.7990017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnl1ke14f/_remote_module_non_scriptable.py 2022-09-27T15:40:21.0593325Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:21.0603331Z 2022-09-27T15:40:21.0603480Z Running tests... 2022-09-27T15:40:21.0603898Z ---------------------------------------------------------------------- 2022-09-27T15:40:21.3003483Z test_bad_module (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1015 2022-09-27T15:40:21.3030191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1016 2022-09-27T15:40:22.3409027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:22.3409672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:22.3467074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:22.3467710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:22.3759375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:22.3760023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:22.3824398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:22.3825049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:22.5004811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9tmvs9q 2022-09-27T15:40:22.5005606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9tmvs9q/_remote_module_non_scriptable.py 2022-09-27T15:40:22.5067816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3nfy9po5 2022-09-27T15:40:22.5069242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3nfy9po5/_remote_module_non_scriptable.py 2022-09-27T15:40:22.7543518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:22.7605363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:22.8954989Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:40:22.8956498Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .") 2022-09-27T15:40:22.8957704Z Traceback (most recent call last): 2022-09-27T15:40:22.8958602Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:40:22.8959300Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:40:22.8960249Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 88, in _create_module 2022-09-27T15:40:22.8961761Z "Expect `module_cls(*args, **kwargs)` returns an instance of , " 2022-09-27T15:40:22.8962917Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of . 2022-09-27T15:40:22.8963557Z 2022-09-27T15:40:22.8963796Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:40:22.8967424Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .")\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 88, in _create_module\n "Expect `module_cls(*args, **kwargs)` returns an instance of , "\nValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .\n') 2022-09-27T15:40:22.8969698Z Traceback (most recent call last): 2022-09-27T15:40:22.8970522Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:40:22.8971238Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:40:22.8971905Z File "/tmp/tmpnl1ke14f/_remote_module_non_scriptable.py", line 47, in _remote_forward 2022-09-27T15:40:22.8972456Z module = module_rref.local_value() 2022-09-27T15:40:22.8973301Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-09-27T15:40:22.8974218Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-09-27T15:40:22.8974824Z ValueError: On WorkerInfo(id=1, name=worker1): 2022-09-27T15:40:22.8975967Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .") 2022-09-27T15:40:22.8976822Z Traceback (most recent call last): 2022-09-27T15:40:22.8977616Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:40:22.8978321Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:40:22.8979231Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 88, in _create_module 2022-09-27T15:40:22.8979884Z "Expect `module_cls(*args, **kwargs)` returns an instance of , " 2022-09-27T15:40:22.8981130Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of . 2022-09-27T15:40:22.8981792Z 2022-09-27T15:40:22.8981801Z 2022-09-27T15:40:22.8983442Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:40:22.8985670Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .") 2022-09-27T15:40:22.8986611Z Traceback (most recent call last): 2022-09-27T15:40:22.8987475Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:40:22.8988188Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:40:22.8989047Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 88, in _create_module 2022-09-27T15:40:22.8989798Z "Expect `module_cls(*args, **kwargs)` returns an instance of , " 2022-09-27T15:40:22.8991111Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of . 2022-09-27T15:40:22.8991806Z 2022-09-27T15:40:22.8991997Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:40:22.8995447Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .")\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 88, in _create_module\n "Expect `module_cls(*args, **kwargs)` returns an instance of , "\nValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .\n') 2022-09-27T15:40:22.8997697Z Traceback (most recent call last): 2022-09-27T15:40:22.8998530Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:40:22.8999218Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:40:22.8999865Z File "/tmp/tmpnl1ke14f/_remote_module_non_scriptable.py", line 47, in _remote_forward 2022-09-27T15:40:22.9000427Z module = module_rref.local_value() 2022-09-27T15:40:22.9005762Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-09-27T15:40:22.9006700Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-09-27T15:40:22.9007319Z ValueError: On WorkerInfo(id=1, name=worker1): 2022-09-27T15:40:22.9008569Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .") 2022-09-27T15:40:22.9009423Z Traceback (most recent call last): 2022-09-27T15:40:22.9010227Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:40:22.9010936Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:40:22.9011858Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 88, in _create_module 2022-09-27T15:40:22.9012593Z "Expect `module_cls(*args, **kwargs)` returns an instance of , " 2022-09-27T15:40:22.9013877Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of . 2022-09-27T15:40:22.9014534Z 2022-09-27T15:40:22.9014543Z 2022-09-27T15:40:23.2072519Z ok (2.147s) 2022-09-27T15:40:23.2072747Z 2022-09-27T15:40:23.2073515Z ---------------------------------------------------------------------- 2022-09-27T15:40:23.2073982Z Ran 1 test in 2.147s 2022-09-27T15:40:23.2074193Z 2022-09-27T15:40:23.2074397Z OK 2022-09-27T15:40:23.2074620Z 2022-09-27T15:40:23.2074706Z Generating XML reports... 2022-09-27T15:40:23.2109673Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154021.xml 2022-09-27T15:40:24.5756259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:24.5756638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:24.5764542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:24.5765028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:24.7316154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqkcoz7x 2022-09-27T15:40:24.7317068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqkcoz7x/_remote_module_non_scriptable.py 2022-09-27T15:40:24.9938530Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:24.9947649Z 2022-09-27T15:40:24.9947740Z Running tests... 2022-09-27T15:40:24.9948985Z ---------------------------------------------------------------------- 2022-09-27T15:40:25.2396851Z test_forward_async (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1132 2022-09-27T15:40:25.2424098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1133 2022-09-27T15:40:26.2708464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:26.2709177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:26.2979036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:26.2979708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:26.3066947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:26.3067572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:26.3323672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:26.3324347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:26.4314339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsge9nza2 2022-09-27T15:40:26.4315222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsge9nza2/_remote_module_non_scriptable.py 2022-09-27T15:40:26.4570815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpliltbwj4 2022-09-27T15:40:26.4571650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpliltbwj4/_remote_module_non_scriptable.py 2022-09-27T15:40:26.6867383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:26.7131060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:26.8573177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsge9nza2/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:26.8574433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpliltbwj4/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:26.8653571Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpsge9nza2/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:27.3468817Z ok (2.352s) 2022-09-27T15:40:27.3469310Z 2022-09-27T15:40:27.3469832Z ---------------------------------------------------------------------- 2022-09-27T15:40:27.3470181Z Ran 1 test in 2.352s 2022-09-27T15:40:27.3470299Z 2022-09-27T15:40:27.3470360Z OK 2022-09-27T15:40:27.3470453Z 2022-09-27T15:40:27.3470551Z Generating XML reports... 2022-09-27T15:40:27.3507749Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154024.xml 2022-09-27T15:40:28.7219780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:28.7220162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:28.7228691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:28.7229054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:28.8844372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbv60jd4h 2022-09-27T15:40:28.8845516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbv60jd4h/_remote_module_non_scriptable.py 2022-09-27T15:40:29.1462175Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:29.1472038Z 2022-09-27T15:40:29.1472141Z Running tests... 2022-09-27T15:40:29.1472707Z ---------------------------------------------------------------------- 2022-09-27T15:40:29.3927014Z test_forward_async_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1249 2022-09-27T15:40:29.3954431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1250 2022-09-27T15:40:30.4320148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:30.4320548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:30.4565796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:30.4566163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:30.4685870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:30.4686238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:30.4936612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:30.4937079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:30.5947071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyzpzo6z 2022-09-27T15:40:30.5948442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyzpzo6z/_remote_module_non_scriptable.py 2022-09-27T15:40:30.6182856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe51qcisl 2022-09-27T15:40:30.6183594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe51qcisl/_remote_module_non_scriptable.py 2022-09-27T15:40:30.8537481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:30.8730268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:31.0140691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe51qcisl/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:31.0141326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyzpzo6z/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:31.0235070Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpe51qcisl/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:31.5003401Z ok (2.353s) 2022-09-27T15:40:31.5003957Z 2022-09-27T15:40:31.5004404Z ---------------------------------------------------------------------- 2022-09-27T15:40:31.5004808Z Ran 1 test in 2.353s 2022-09-27T15:40:31.5004987Z 2022-09-27T15:40:31.5005077Z OK 2022-09-27T15:40:31.5005219Z 2022-09-27T15:40:31.5005365Z Generating XML reports... 2022-09-27T15:40:31.5041167Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154029.xml 2022-09-27T15:40:32.8857998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:32.8858370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:32.8865595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:32.8866120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:33.0444237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_rhgwsk 2022-09-27T15:40:33.0445868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_rhgwsk/_remote_module_non_scriptable.py 2022-09-27T15:40:33.3083252Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:33.3093899Z 2022-09-27T15:40:33.3094320Z Running tests... 2022-09-27T15:40:33.3094721Z ---------------------------------------------------------------------- 2022-09-27T15:40:33.5530770Z test_forward_sync (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1370 2022-09-27T15:40:33.5558038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1371 2022-09-27T15:40:34.6033617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:34.6033995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:34.6380932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:34.6381289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:34.6422331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:34.6422672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:34.6789585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:34.6789948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:34.7644438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0b77dhp5 2022-09-27T15:40:34.7645447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0b77dhp5/_remote_module_non_scriptable.py 2022-09-27T15:40:34.8045989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5suvv5y 2022-09-27T15:40:34.8047512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5suvv5y/_remote_module_non_scriptable.py 2022-09-27T15:40:35.0193876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:35.0597453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:35.1980554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5suvv5y/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:35.1981621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0b77dhp5/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:35.2059667Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpz5suvv5y/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:35.6605592Z ok (2.351s) 2022-09-27T15:40:35.6605758Z 2022-09-27T15:40:35.6606119Z ---------------------------------------------------------------------- 2022-09-27T15:40:35.6606396Z Ran 1 test in 2.351s 2022-09-27T15:40:35.6606515Z 2022-09-27T15:40:35.6606576Z OK 2022-09-27T15:40:35.6606671Z 2022-09-27T15:40:35.6606765Z Generating XML reports... 2022-09-27T15:40:35.6642545Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154033.xml 2022-09-27T15:40:37.0157497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:37.0158114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:37.0166410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:37.0167065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:37.1752457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lx2st_n 2022-09-27T15:40:37.1753465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lx2st_n/_remote_module_non_scriptable.py 2022-09-27T15:40:37.4366434Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:37.4376267Z 2022-09-27T15:40:37.4376412Z Running tests... 2022-09-27T15:40:37.4377003Z ---------------------------------------------------------------------- 2022-09-27T15:40:37.6814634Z test_forward_sync_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1491 2022-09-27T15:40:37.6842133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1492 2022-09-27T15:40:38.7086277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:38.7086738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:38.7224061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:38.7224418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:38.7439269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:38.7439636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:38.7580239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:38.7580626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:38.8690124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj26x39ts 2022-09-27T15:40:38.8690894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj26x39ts/_remote_module_non_scriptable.py 2022-09-27T15:40:38.8823390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjwb6nj9 2022-09-27T15:40:38.8825023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjwb6nj9/_remote_module_non_scriptable.py 2022-09-27T15:40:39.1215540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:39.1362228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:39.2763551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjwb6nj9/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:39.2765093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj26x39ts/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:39.2818081Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpj26x39ts/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:39.6885362Z ok (2.251s) 2022-09-27T15:40:39.6885683Z 2022-09-27T15:40:39.6886122Z ---------------------------------------------------------------------- 2022-09-27T15:40:39.6886388Z Ran 1 test in 2.251s 2022-09-27T15:40:39.6886505Z 2022-09-27T15:40:39.6886566Z OK 2022-09-27T15:40:39.6886645Z 2022-09-27T15:40:39.6886742Z Generating XML reports... 2022-09-27T15:40:39.6922398Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154037.xml 2022-09-27T15:40:41.0409085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:41.0409519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:41.0417388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:41.0417767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:41.1978265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1eu57bo 2022-09-27T15:40:41.1979985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1eu57bo/_remote_module_non_scriptable.py 2022-09-27T15:40:41.4624765Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:41.4633728Z 2022-09-27T15:40:41.4633845Z Running tests... 2022-09-27T15:40:41.4635130Z ---------------------------------------------------------------------- 2022-09-27T15:40:41.7091955Z test_forward_with_kwargs (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1612 2022-09-27T15:40:41.7120223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1613 2022-09-27T15:40:42.7610851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:42.7886685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:42.7887494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:42.7887839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:42.7955944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:42.7956432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:42.8230445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:42.8230990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:42.9205805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71y29pzp 2022-09-27T15:40:42.9206402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71y29pzp/_remote_module_non_scriptable.py 2022-09-27T15:40:42.9476811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpai4q4vmu 2022-09-27T15:40:42.9477504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpai4q4vmu/_remote_module_non_scriptable.py 2022-09-27T15:40:43.1730595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:43.1996725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:43.7167687Z ok (2.253s) 2022-09-27T15:40:43.7167934Z 2022-09-27T15:40:43.7168459Z ---------------------------------------------------------------------- 2022-09-27T15:40:43.7168757Z Ran 1 test in 2.253s 2022-09-27T15:40:43.7169120Z 2022-09-27T15:40:43.7169183Z OK 2022-09-27T15:40:43.7169274Z 2022-09-27T15:40:43.7169370Z Generating XML reports... 2022-09-27T15:40:43.7206014Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154041.xml 2022-09-27T15:40:45.0847504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:45.0847885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:45.0855525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:45.0855983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:45.2417273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgate70i 2022-09-27T15:40:45.2418387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgate70i/_remote_module_non_scriptable.py 2022-09-27T15:40:45.5030255Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:45.5040025Z 2022-09-27T15:40:45.5040354Z Running tests... 2022-09-27T15:40:45.5048877Z ---------------------------------------------------------------------- 2022-09-27T15:40:45.7440585Z test_get_module_rref (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1729 2022-09-27T15:40:45.7466583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1730 2022-09-27T15:40:46.7628365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:46.7629094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:46.7897029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:46.7897671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:46.7979922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:46.7980563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:46.8224733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:46.8225405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:46.9220239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpke74kt_6 2022-09-27T15:40:46.9221403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpke74kt_6/_remote_module_non_scriptable.py 2022-09-27T15:40:46.9468875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vog1bs0 2022-09-27T15:40:46.9470149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vog1bs0/_remote_module_non_scriptable.py 2022-09-27T15:40:47.1732922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:47.1993216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:47.7510035Z ok (2.247s) 2022-09-27T15:40:47.7510195Z 2022-09-27T15:40:47.7510507Z ---------------------------------------------------------------------- 2022-09-27T15:40:47.7510841Z Ran 1 test in 2.247s 2022-09-27T15:40:47.7511051Z 2022-09-27T15:40:47.7511164Z OK 2022-09-27T15:40:47.7511371Z 2022-09-27T15:40:47.7511534Z Generating XML reports... 2022-09-27T15:40:47.7546698Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154045.xml 2022-09-27T15:40:49.1006775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:49.1007420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:49.1015370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:49.1015850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:49.2575167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49i738e5 2022-09-27T15:40:49.2575893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49i738e5/_remote_module_non_scriptable.py 2022-09-27T15:40:49.5180220Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:49.5189231Z 2022-09-27T15:40:49.5189344Z Running tests... 2022-09-27T15:40:49.5190280Z ---------------------------------------------------------------------- 2022-09-27T15:40:49.7592818Z test_remote_module_py_pickle_not_supported (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1846 2022-09-27T15:40:49.7620045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1847 2022-09-27T15:40:50.7846860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:50.7847508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:50.7863074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:50.7863713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:50.8210326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:50.8211078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:50.8226985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:50.8227403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:50.9457698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7qbzl7y 2022-09-27T15:40:50.9459018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7qbzl7y/_remote_module_non_scriptable.py 2022-09-27T15:40:50.9475000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5ak9unt 2022-09-27T15:40:50.9477279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5ak9unt/_remote_module_non_scriptable.py 2022-09-27T15:40:51.1974215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:51.2015884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:51.6661944Z ok (2.147s) 2022-09-27T15:40:51.6662222Z 2022-09-27T15:40:51.6662689Z ---------------------------------------------------------------------- 2022-09-27T15:40:51.6662936Z Ran 1 test in 2.147s 2022-09-27T15:40:51.6663069Z 2022-09-27T15:40:51.6663128Z OK 2022-09-27T15:40:51.6663217Z 2022-09-27T15:40:51.6663318Z Generating XML reports... 2022-09-27T15:40:51.6698899Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154049.xml 2022-09-27T15:40:53.0394373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:53.0394915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:53.0402574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:53.0403100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:53.1964829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmduw66hg 2022-09-27T15:40:53.1966432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmduw66hg/_remote_module_non_scriptable.py 2022-09-27T15:40:53.4571524Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:53.4580874Z 2022-09-27T15:40:53.4581119Z Running tests... 2022-09-27T15:40:53.4581722Z ---------------------------------------------------------------------- 2022-09-27T15:40:53.6994209Z test_remote_module_py_pickle_not_supported_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1963 2022-09-27T15:40:53.7022751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1964 2022-09-27T15:40:54.7247472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:54.7248216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:54.7316085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:54.7316584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:54.7601298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:54.7601952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:54.7666122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:54.7666530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:54.8844732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqy2ypjh0 2022-09-27T15:40:54.8846108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqy2ypjh0/_remote_module_non_scriptable.py 2022-09-27T15:40:54.8920022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1j7x7_z 2022-09-27T15:40:54.8921443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1j7x7_z/_remote_module_non_scriptable.py 2022-09-27T15:40:55.1395530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:55.1468374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:55.2566211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqy2ypjh0/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:55.2567462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1j7x7_z/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:55.2637002Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmps1j7x7_z/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T15:40:55.7064785Z ok (2.248s) 2022-09-27T15:40:55.7065032Z 2022-09-27T15:40:55.7065516Z ---------------------------------------------------------------------- 2022-09-27T15:40:55.7065782Z Ran 1 test in 2.248s 2022-09-27T15:40:55.7065893Z 2022-09-27T15:40:55.7065954Z OK 2022-09-27T15:40:55.7066046Z 2022-09-27T15:40:55.7066141Z Generating XML reports... 2022-09-27T15:40:55.7100924Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154053.xml 2022-09-27T15:40:57.0839412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:57.0840054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:57.0848287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:57.0848880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:57.2430541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakgh5rk9 2022-09-27T15:40:57.2431568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakgh5rk9/_remote_module_non_scriptable.py 2022-09-27T15:40:57.5039496Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:40:57.5048862Z 2022-09-27T15:40:57.5048988Z Running tests... 2022-09-27T15:40:57.5049426Z ---------------------------------------------------------------------- 2022-09-27T15:40:57.7476379Z test_remote_parameters (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2080 2022-09-27T15:40:57.7503268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2081 2022-09-27T15:40:58.7798016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:58.7798508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:58.7799044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:40:58.7799385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:40:58.8150194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:58.8150569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:58.8154358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:40:58.8155032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:40:58.9417082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gacd6em 2022-09-27T15:40:58.9417825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm47_d7xj 2022-09-27T15:40:58.9418417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gacd6em/_remote_module_non_scriptable.py 2022-09-27T15:40:58.9418823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm47_d7xj/_remote_module_non_scriptable.py 2022-09-27T15:40:59.1954326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:40:59.1954936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:40:59.6544663Z ok (2.149s) 2022-09-27T15:40:59.6544882Z 2022-09-27T15:40:59.6545318Z ---------------------------------------------------------------------- 2022-09-27T15:40:59.6545685Z Ran 1 test in 2.149s 2022-09-27T15:40:59.6545878Z 2022-09-27T15:40:59.6545975Z OK 2022-09-27T15:40:59.6546114Z 2022-09-27T15:40:59.6546262Z Generating XML reports... 2022-09-27T15:40:59.6583002Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154057.xml 2022-09-27T15:41:01.0345144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:01.0345530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:01.0353265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:01.0353635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:01.1918819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxf5jh3xq 2022-09-27T15:41:01.1919760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxf5jh3xq/_remote_module_non_scriptable.py 2022-09-27T15:41:01.4572757Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:41:01.4582303Z 2022-09-27T15:41:01.4582403Z Running tests... 2022-09-27T15:41:01.4582777Z ---------------------------------------------------------------------- 2022-09-27T15:41:01.7064782Z test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2197 2022-09-27T15:41:01.7092111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2198 2022-09-27T15:41:02.7676356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:02.7676756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:02.7981357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:02.7981738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:02.8029973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:02.8030648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:02.8330181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:02.8330820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:02.9279859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fx0250b 2022-09-27T15:41:02.9280650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fx0250b/_remote_module_non_scriptable.py 2022-09-27T15:41:02.9586166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvk_thhg 2022-09-27T15:41:02.9587231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvk_thhg/_remote_module_non_scriptable.py 2022-09-27T15:41:03.1822544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:41:03.2142641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:41:03.3643199Z The new attribute ``new_attr`` of RemoteModule is ignored during RPC pickling. To pickle this attribute, please add it to ``_REMOTE_MODULE_PICKLED_ATTRIBUTES``. Otherwise, please explicitly add it to ``_REMOTE_MODULE_ATTRIBUTES_IGNORE_FOR_PICKLING``. 2022-09-27T15:41:03.7136823Z ok (2.255s) 2022-09-27T15:41:03.7137118Z 2022-09-27T15:41:03.7137602Z ---------------------------------------------------------------------- 2022-09-27T15:41:03.7137838Z Ran 1 test in 2.255s 2022-09-27T15:41:03.7137955Z 2022-09-27T15:41:03.7138016Z OK 2022-09-27T15:41:03.7138109Z 2022-09-27T15:41:03.7138205Z Generating XML reports... 2022-09-27T15:41:03.7172788Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154101.xml 2022-09-27T15:41:05.0829811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:05.0830202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:05.0838142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:05.0838649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:05.2399146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy65or50d 2022-09-27T15:41:05.2400208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy65or50d/_remote_module_non_scriptable.py 2022-09-27T15:41:05.5022101Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:41:05.5031802Z 2022-09-27T15:41:05.5031935Z Running tests... 2022-09-27T15:41:05.5032411Z ---------------------------------------------------------------------- 2022-09-27T15:41:05.7493238Z test_train_eval (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2314 2022-09-27T15:41:05.7521149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2315 2022-09-27T15:41:06.7937521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:06.7938487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:06.7957262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:06.7957892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:06.8301876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:06.8302631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:06.8324662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:06.8325191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:06.9566421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0nyba118 2022-09-27T15:41:06.9567272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0nyba118/_remote_module_non_scriptable.py 2022-09-27T15:41:06.9715133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbr10422 2022-09-27T15:41:06.9716018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbr10422/_remote_module_non_scriptable.py 2022-09-27T15:41:07.2107467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:41:07.2245726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:41:07.6566202Z ok (2.153s) 2022-09-27T15:41:07.6566474Z 2022-09-27T15:41:07.6566883Z ---------------------------------------------------------------------- 2022-09-27T15:41:07.6567154Z Ran 1 test in 2.153s 2022-09-27T15:41:07.6567273Z 2022-09-27T15:41:07.6567320Z OK 2022-09-27T15:41:07.6567429Z 2022-09-27T15:41:07.6567527Z Generating XML reports... 2022-09-27T15:41:07.6603388Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154105.xml 2022-09-27T15:41:09.0562996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:09.0563413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:09.0570881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:09.0571282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:09.2126088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnx_9n6sx 2022-09-27T15:41:09.2127241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnx_9n6sx/_remote_module_non_scriptable.py 2022-09-27T15:41:09.4763246Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:41:09.4773195Z 2022-09-27T15:41:09.4773614Z Running tests... 2022-09-27T15:41:09.4774022Z ---------------------------------------------------------------------- 2022-09-27T15:41:09.7281888Z test_unsupported_methods (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2431 2022-09-27T15:41:09.7309002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2432 2022-09-27T15:41:10.7906255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:10.7906647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:10.8021500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:10.8021846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:10.8250380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:10.8250862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:10.8383431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:10.8383803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:10.9494923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhh50_so 2022-09-27T15:41:10.9495729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhh50_so/_remote_module_non_scriptable.py 2022-09-27T15:41:10.9619101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4s2zmcqg 2022-09-27T15:41:10.9621269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4s2zmcqg/_remote_module_non_scriptable.py 2022-09-27T15:41:11.2012717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:41:11.2153184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:41:11.7353307Z ok (2.258s) 2022-09-27T15:41:11.7353534Z 2022-09-27T15:41:11.7353982Z ---------------------------------------------------------------------- 2022-09-27T15:41:11.7354401Z Ran 1 test in 2.258s 2022-09-27T15:41:11.7354755Z 2022-09-27T15:41:11.7354858Z OK 2022-09-27T15:41:11.7355005Z 2022-09-27T15:41:11.7355152Z Generating XML reports... 2022-09-27T15:41:11.7391758Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154109.xml 2022-09-27T15:41:13.1399654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:13.1400044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:13.1408189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:13.1408594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:13.2998638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjndw1sk9 2022-09-27T15:41:13.2999717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjndw1sk9/_remote_module_non_scriptable.py 2022-09-27T15:41:13.5640375Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:41:13.5650599Z 2022-09-27T15:41:13.5650703Z Running tests... 2022-09-27T15:41:13.5651191Z ---------------------------------------------------------------------- 2022-09-27T15:41:13.8098322Z test_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2548 2022-09-27T15:41:13.8125439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2549 2022-09-27T15:41:13.8152558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2550 2022-09-27T15:41:13.8181825Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2551 2022-09-27T15:41:14.8639607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:14.8640260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:14.8811618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:14.8812281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:14.8912649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:14.8913302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:14.8997477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:14.8998144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:14.9180420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:14.9181068Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:14.9218488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:14.9219194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:14.9274167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:14.9274957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:14.9572234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:14.9572894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:15.0254111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpod7wekql 2022-09-27T15:41:15.0254897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpod7wekql/_remote_module_non_scriptable.py 2022-09-27T15:41:15.0448207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpne0ueeo_ 2022-09-27T15:41:15.0449154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpne0ueeo_/_remote_module_non_scriptable.py 2022-09-27T15:41:15.0535673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ofp3gcx 2022-09-27T15:41:15.0537125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ofp3gcx/_remote_module_non_scriptable.py 2022-09-27T15:41:15.0819706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_rlnjn4f 2022-09-27T15:41:15.0820850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_rlnjn4f/_remote_module_non_scriptable.py 2022-09-27T15:41:15.2827852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:41:15.3131435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:41:15.3208304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:41:15.3393524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:41:16.0239643Z ok (2.459s) 2022-09-27T15:41:16.0239925Z 2022-09-27T15:41:16.0240381Z ---------------------------------------------------------------------- 2022-09-27T15:41:16.0240634Z Ran 1 test in 2.459s 2022-09-27T15:41:16.0240735Z 2022-09-27T15:41:16.0240796Z OK 2022-09-27T15:41:16.0240888Z 2022-09-27T15:41:16.0240981Z Generating XML reports... 2022-09-27T15:41:16.0275655Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154113.xml 2022-09-27T15:41:17.3831710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:17.3832340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:17.3840672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:17.3841314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:17.5408843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2k970b9 2022-09-27T15:41:17.5409795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2k970b9/_remote_module_non_scriptable.py 2022-09-27T15:41:17.8012969Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:41:17.8023051Z 2022-09-27T15:41:17.8023402Z Running tests... 2022-09-27T15:41:17.8023749Z ---------------------------------------------------------------------- 2022-09-27T15:41:18.0420359Z test_add_done_callback (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2771 2022-09-27T15:41:18.0446752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2772 2022-09-27T15:41:18.0474828Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2773 2022-09-27T15:41:18.0504211Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2774 2022-09-27T15:41:19.1741464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:19.1742085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:19.2087584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:19.2088198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:19.2101362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:19.2102007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:19.2448591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:19.2448965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:19.2595780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:19.2596331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:19.2749803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:19.2750416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:19.2831617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:19.2832261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:19.3113444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:19.3114104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:19.3404583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6xq002m 2022-09-27T15:41:19.3405400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6xq002m/_remote_module_non_scriptable.py 2022-09-27T15:41:19.3888560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphw4gm8jt 2022-09-27T15:41:19.3889368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphw4gm8jt/_remote_module_non_scriptable.py 2022-09-27T15:41:19.4092407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mazeqly 2022-09-27T15:41:19.4093126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mazeqly/_remote_module_non_scriptable.py 2022-09-27T15:41:19.4370629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmnmbzk1 2022-09-27T15:41:19.4371433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmnmbzk1/_remote_module_non_scriptable.py 2022-09-27T15:41:19.5992770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:41:19.6465929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:41:19.6633013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:41:19.6906706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:41:20.3563356Z ok (2.554s) 2022-09-27T15:41:20.3564003Z 2022-09-27T15:41:20.3564402Z ---------------------------------------------------------------------- 2022-09-27T15:41:20.3564714Z Ran 1 test in 2.554s 2022-09-27T15:41:20.3564830Z 2022-09-27T15:41:20.3564893Z OK 2022-09-27T15:41:20.3564983Z 2022-09-27T15:41:20.3565077Z Generating XML reports... 2022-09-27T15:41:20.3602500Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154117.xml 2022-09-27T15:41:21.7413937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:21.7414321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:21.7421675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:21.7422026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:21.8991733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpia09rb_8 2022-09-27T15:41:21.8992631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpia09rb_8/_remote_module_non_scriptable.py 2022-09-27T15:41:22.1625343Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:41:22.1635462Z 2022-09-27T15:41:22.1635877Z Running tests... 2022-09-27T15:41:22.1636277Z ---------------------------------------------------------------------- 2022-09-27T15:41:22.4143207Z test_add_with_id (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2994 2022-09-27T15:41:22.4169976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2995 2022-09-27T15:41:22.4197719Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2996 2022-09-27T15:41:22.4227887Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2997 2022-09-27T15:41:23.5777606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:23.5778304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:23.6015889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:23.6016388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:23.6165994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:23.6166559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:23.6396373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:23.6397066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:23.6476552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:23.6477204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:23.6502756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:23.6503370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:23.6842477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:23.6843165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:23.6865873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:23.7440199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:23.7440974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl4y0h8x7 2022-09-27T15:41:23.7441982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl4y0h8x7/_remote_module_non_scriptable.py 2022-09-27T15:41:23.7669039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49zxgtj3 2022-09-27T15:41:23.7670094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49zxgtj3/_remote_module_non_scriptable.py 2022-09-27T15:41:23.8110596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8fymy2g 2022-09-27T15:41:23.8111603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8fymy2g/_remote_module_non_scriptable.py 2022-09-27T15:41:23.8127953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwuqh5dh7 2022-09-27T15:41:23.8130290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwuqh5dh7/_remote_module_non_scriptable.py 2022-09-27T15:41:24.0009887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:41:24.0251822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:41:24.0673754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:41:24.0704344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:41:24.7284966Z ok (2.565s) 2022-09-27T15:41:24.7285232Z 2022-09-27T15:41:24.7285752Z ---------------------------------------------------------------------- 2022-09-27T15:41:24.7286113Z Ran 1 test in 2.565s 2022-09-27T15:41:24.7286232Z 2022-09-27T15:41:24.7286293Z OK 2022-09-27T15:41:24.7286384Z 2022-09-27T15:41:24.7286479Z Generating XML reports... 2022-09-27T15:41:24.7321538Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154122.xml 2022-09-27T15:41:26.1164315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:26.1164716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:26.1172955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:26.1173460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:26.2763332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_ri_9fb 2022-09-27T15:41:26.2764512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_ri_9fb/_remote_module_non_scriptable.py 2022-09-27T15:41:26.5378306Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:41:26.5387806Z 2022-09-27T15:41:26.5388118Z Running tests... 2022-09-27T15:41:26.5388695Z ---------------------------------------------------------------------- 2022-09-27T15:41:26.7836487Z test_all_gather (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3217 2022-09-27T15:41:26.7864877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3218 2022-09-27T15:41:26.7892934Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3219 2022-09-27T15:41:26.7922359Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3220 2022-09-27T15:41:27.8901638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:27.8902317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:27.8922002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:27.9052867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:27.9053644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:27.9054467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:27.9122611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:27.9123392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:27.9299879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:27.9300399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:27.9321339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:27.9321927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:27.9455398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:27.9456027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:27.9512655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:27.9513309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:28.0561079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbkk9t0g 2022-09-27T15:41:28.0561775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbkk9t0g/_remote_module_non_scriptable.py 2022-09-27T15:41:28.0582549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposhuu6rp 2022-09-27T15:41:28.0583639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposhuu6rp/_remote_module_non_scriptable.py 2022-09-27T15:41:28.0722855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkbe42h6n 2022-09-27T15:41:28.0723791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkbe42h6n/_remote_module_non_scriptable.py 2022-09-27T15:41:28.0762592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5a92msdn 2022-09-27T15:41:28.0764356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5a92msdn/_remote_module_non_scriptable.py 2022-09-27T15:41:28.3105507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:41:28.3150452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:41:28.3306594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:41:28.3315743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:41:28.8977146Z ok (2.359s) 2022-09-27T15:41:28.8977374Z 2022-09-27T15:41:28.8977779Z ---------------------------------------------------------------------- 2022-09-27T15:41:28.8978096Z Ran 1 test in 2.359s 2022-09-27T15:41:28.8978310Z 2022-09-27T15:41:28.8980115Z OK 2022-09-27T15:41:28.8980356Z 2022-09-27T15:41:28.8980555Z Generating XML reports... 2022-09-27T15:41:28.9017012Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154126.xml 2022-09-27T15:41:30.2744641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:30.2745032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:30.2753272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:30.2753651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:30.4335792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2et_3pgm 2022-09-27T15:41:30.4337213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2et_3pgm/_remote_module_non_scriptable.py 2022-09-27T15:41:30.6949202Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:41:30.6958839Z 2022-09-27T15:41:30.6958942Z Running tests... 2022-09-27T15:41:30.6959508Z ---------------------------------------------------------------------- 2022-09-27T15:41:30.9376849Z test_all_gather_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3441 2022-09-27T15:41:30.9404301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3442 2022-09-27T15:41:30.9431558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3443 2022-09-27T15:41:30.9460460Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3444 2022-09-27T15:41:31.9960646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:31.9961249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:32.0220041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:32.0220702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:32.0309171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:32.0309816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:32.0435385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:32.0436073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:32.0550502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:32.0551040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:32.0589531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:32.0590110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:32.0816982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:32.0817671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:32.0921089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:32.0921746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:32.1586137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbul2i2oa 2022-09-27T15:41:32.1587222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbul2i2oa/_remote_module_non_scriptable.py 2022-09-27T15:41:32.1864653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7vn3z0dx 2022-09-27T15:41:32.1866099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7vn3z0dx/_remote_module_non_scriptable.py 2022-09-27T15:41:32.2090749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3nl4qrp1 2022-09-27T15:41:32.2092236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3nl4qrp1/_remote_module_non_scriptable.py 2022-09-27T15:41:32.2188030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwogul02f 2022-09-27T15:41:32.2188702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwogul02f/_remote_module_non_scriptable.py 2022-09-27T15:41:32.4165831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:41:32.4468714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:41:32.4704614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:41:32.4761677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:41:41.7183964Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #5 to worker0, as the agent is shutting down 2022-09-27T15:41:42.0656537Z ok (11.369s) 2022-09-27T15:41:42.0656766Z 2022-09-27T15:41:42.0657208Z ---------------------------------------------------------------------- 2022-09-27T15:41:42.0657615Z Ran 1 test in 11.370s 2022-09-27T15:41:42.0657790Z 2022-09-27T15:41:42.0657877Z OK 2022-09-27T15:41:42.0658025Z 2022-09-27T15:41:42.0658152Z Generating XML reports... 2022-09-27T15:41:42.0694064Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154130.xml 2022-09-27T15:41:43.4508806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:43.4509185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:43.4517245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:43.4517670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:43.6079856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2qja38r 2022-09-27T15:41:43.6080665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2qja38r/_remote_module_non_scriptable.py 2022-09-27T15:41:43.8682079Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:41:43.8691861Z 2022-09-27T15:41:43.8692093Z Running tests... 2022-09-27T15:41:43.8692703Z ---------------------------------------------------------------------- 2022-09-27T15:41:44.1118871Z test_async_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3664 2022-09-27T15:41:44.1146801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3665 2022-09-27T15:41:44.1175821Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3666 2022-09-27T15:41:44.1206677Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3667 2022-09-27T15:41:45.2747569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:45.2748251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:45.2749040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:45.2749643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:45.3064184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:45.3064857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:45.3113232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:45.3113889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:45.3124581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:45.3125102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:45.3128354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:45.3128863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:45.3436033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:45.3436648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:45.3484111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:45.3485018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:45.4394672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74mw038o 2022-09-27T15:41:45.4396276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74mw038o/_remote_module_non_scriptable.py 2022-09-27T15:41:45.4575404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbzqw_wlu 2022-09-27T15:41:45.4576530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbzqw_wlu/_remote_module_non_scriptable.py 2022-09-27T15:41:45.4713077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyywzy1tn 2022-09-27T15:41:45.4714324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyywzy1tn/_remote_module_non_scriptable.py 2022-09-27T15:41:45.4757319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxxi0upp 2022-09-27T15:41:45.4758614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxxi0upp/_remote_module_non_scriptable.py 2022-09-27T15:41:45.6935808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:41:45.7161036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:41:45.7277040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:41:45.7328517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:41:46.4264073Z ok (2.557s) 2022-09-27T15:41:46.4264299Z 2022-09-27T15:41:46.4264606Z ---------------------------------------------------------------------- 2022-09-27T15:41:46.4264843Z Ran 1 test in 2.557s 2022-09-27T15:41:46.4264960Z 2022-09-27T15:41:46.4265019Z OK 2022-09-27T15:41:46.4265115Z 2022-09-27T15:41:46.4265211Z Generating XML reports... 2022-09-27T15:41:46.4300898Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154143.xml 2022-09-27T15:41:47.8097378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:47.8098038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:47.8107986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:47.8108602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:47.9730073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3jqwa1q 2022-09-27T15:41:47.9731046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3jqwa1q/_remote_module_non_scriptable.py 2022-09-27T15:41:48.2339793Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:41:48.2349848Z 2022-09-27T15:41:48.2350141Z Running tests... 2022-09-27T15:41:48.2350792Z ---------------------------------------------------------------------- 2022-09-27T15:41:48.4792927Z test_async_class_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3887 2022-09-27T15:41:48.4820777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3888 2022-09-27T15:41:48.4848752Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3889 2022-09-27T15:41:48.4878715Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3890 2022-09-27T15:41:49.5890253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:49.5890858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:49.5898495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:49.5899388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:49.5900610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:49.5901339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:49.6105783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:49.6106313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:49.6276988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:49.6277700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:49.6285492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:49.6286106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:49.6291591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:49.6292177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:49.6495334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:49.6495998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:49.7550056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4btq9cq1 2022-09-27T15:41:49.7550801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4btq9cq1/_remote_module_non_scriptable.py 2022-09-27T15:41:49.7557198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfgzbb88 2022-09-27T15:41:49.7559537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfgzbb88/_remote_module_non_scriptable.py 2022-09-27T15:41:49.7560628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppleruqcx 2022-09-27T15:41:49.7563456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppleruqcx/_remote_module_non_scriptable.py 2022-09-27T15:41:49.7768682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbq8e_0f_ 2022-09-27T15:41:49.7769917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbq8e_0f_/_remote_module_non_scriptable.py 2022-09-27T15:41:50.0121288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:41:50.0121993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:41:50.0135058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:41:50.0359213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:41:50.6935660Z ok (2.458s) 2022-09-27T15:41:50.6935928Z 2022-09-27T15:41:50.6936424Z ---------------------------------------------------------------------- 2022-09-27T15:41:50.6936695Z Ran 1 test in 2.458s 2022-09-27T15:41:50.6936817Z 2022-09-27T15:41:50.6936876Z OK 2022-09-27T15:41:50.6936966Z 2022-09-27T15:41:50.6937062Z Generating XML reports... 2022-09-27T15:41:50.6973349Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154148.xml 2022-09-27T15:41:52.0682913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:52.0683277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:52.0691093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:52.0691493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:52.2271620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8b_wkqm 2022-09-27T15:41:52.2272630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8b_wkqm/_remote_module_non_scriptable.py 2022-09-27T15:41:52.4905852Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:41:52.4916102Z 2022-09-27T15:41:52.4916271Z Running tests... 2022-09-27T15:41:52.4916703Z ---------------------------------------------------------------------- 2022-09-27T15:41:52.7403108Z test_async_class_method_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4110 2022-09-27T15:41:52.7432367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4111 2022-09-27T15:41:52.7461523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4112 2022-09-27T15:41:52.7491495Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4113 2022-09-27T15:41:53.8069488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:53.8070212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:53.8435230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:53.8435822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:53.8650069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:53.8650738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:53.8680707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:53.8681342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:53.8829026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:53.8829706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:53.9040588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:53.9041269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:53.9074461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:53.9075305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:53.9212603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:53.9213265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:53.9714295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxu5_ktbg 2022-09-27T15:41:53.9715201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxu5_ktbg/_remote_module_non_scriptable.py 2022-09-27T15:41:54.0305267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76q32ox1 2022-09-27T15:41:54.0306016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76q32ox1/_remote_module_non_scriptable.py 2022-09-27T15:41:54.0334603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jnsm597 2022-09-27T15:41:54.0335548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jnsm597/_remote_module_non_scriptable.py 2022-09-27T15:41:54.0483154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmuyn_xui 2022-09-27T15:41:54.0484341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmuyn_xui/_remote_module_non_scriptable.py 2022-09-27T15:41:54.2284788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:41:54.2863916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:41:54.2899902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:41:54.3082899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:41:54.9548132Z ok (2.463s) 2022-09-27T15:41:54.9548357Z 2022-09-27T15:41:54.9548844Z ---------------------------------------------------------------------- 2022-09-27T15:41:54.9549313Z Ran 1 test in 2.463s 2022-09-27T15:41:54.9549486Z 2022-09-27T15:41:54.9549534Z OK 2022-09-27T15:41:54.9549626Z 2022-09-27T15:41:54.9549726Z Generating XML reports... 2022-09-27T15:41:54.9584788Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154152.xml 2022-09-27T15:41:56.3385963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:56.3386384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:56.3395219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:56.3395628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:56.4967095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpam32o8ff 2022-09-27T15:41:56.4967829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpam32o8ff/_remote_module_non_scriptable.py 2022-09-27T15:41:56.7609345Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:41:56.7619718Z 2022-09-27T15:41:56.7619841Z Running tests... 2022-09-27T15:41:56.7620485Z ---------------------------------------------------------------------- 2022-09-27T15:41:57.0056754Z test_async_class_rref_proxy (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4333 2022-09-27T15:41:57.0084191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4334 2022-09-27T15:41:57.0112732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4335 2022-09-27T15:41:57.0142730Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4336 2022-09-27T15:41:58.1417332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:58.1417994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:58.1461692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:58.1462347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:58.1588384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:58.1588897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:58.1657144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:41:58.1657677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:41:58.1792577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:58.1793301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:58.1845460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:58.1846139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:58.1973737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:58.1974357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:58.2037013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:41:58.2037770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:41:58.3061306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwi2ldq4 2022-09-27T15:41:58.3062097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwi2ldq4/_remote_module_non_scriptable.py 2022-09-27T15:41:58.3111339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr53_6erk 2022-09-27T15:41:58.3112501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr53_6erk/_remote_module_non_scriptable.py 2022-09-27T15:41:58.3243285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvmj6_uv 2022-09-27T15:41:58.3244321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvmj6_uv/_remote_module_non_scriptable.py 2022-09-27T15:41:58.3287752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrpwquch 2022-09-27T15:41:58.3289288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrpwquch/_remote_module_non_scriptable.py 2022-09-27T15:41:58.5652239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:41:58.5694979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:41:58.5857888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:41:58.5870332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:41:59.3200338Z ok (2.558s) 2022-09-27T15:41:59.3200603Z 2022-09-27T15:41:59.3201111Z ---------------------------------------------------------------------- 2022-09-27T15:41:59.3201395Z Ran 1 test in 2.558s 2022-09-27T15:41:59.3201509Z 2022-09-27T15:41:59.3201570Z OK 2022-09-27T15:41:59.3201648Z 2022-09-27T15:41:59.3201757Z Generating XML reports... 2022-09-27T15:41:59.3236729Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154156.xml 2022-09-27T15:42:00.6899255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:00.6899805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:00.6908083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:00.6908703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:00.8509595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zm1g_bs 2022-09-27T15:42:00.8510270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zm1g_bs/_remote_module_non_scriptable.py 2022-09-27T15:42:01.1133249Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:01.1143422Z 2022-09-27T15:42:01.1143749Z Running tests... 2022-09-27T15:42:01.1144338Z ---------------------------------------------------------------------- 2022-09-27T15:42:01.3587566Z test_async_class_rref_proxy_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4556 2022-09-27T15:42:01.3616232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4557 2022-09-27T15:42:01.3645571Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4558 2022-09-27T15:42:01.3675783Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4559 2022-09-27T15:42:02.4372054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:02.4372665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:02.4438478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:02.4439189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:02.4745057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:02.4745689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:02.4813255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:02.4813867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:02.4853140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:02.4853734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:02.4989291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:02.4989888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:02.5221102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:02.5221748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:02.5365799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:02.5366455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:02.6014457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5_oc9e2 2022-09-27T15:42:02.6015362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5_oc9e2/_remote_module_non_scriptable.py 2022-09-27T15:42:02.6083159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg57pfqd1 2022-09-27T15:42:02.6084717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg57pfqd1/_remote_module_non_scriptable.py 2022-09-27T15:42:02.6473694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj84renyd 2022-09-27T15:42:02.6474927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj84renyd/_remote_module_non_scriptable.py 2022-09-27T15:42:02.6625668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3kvaw7j 2022-09-27T15:42:02.6626465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3kvaw7j/_remote_module_non_scriptable.py 2022-09-27T15:42:02.8587090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:42:02.8671328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:42:02.9050657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:42:02.9229653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:42:03.5731867Z ok (2.458s) 2022-09-27T15:42:03.5732149Z 2022-09-27T15:42:03.5732638Z ---------------------------------------------------------------------- 2022-09-27T15:42:03.5732876Z Ran 1 test in 2.459s 2022-09-27T15:42:03.5732996Z 2022-09-27T15:42:03.5733057Z OK 2022-09-27T15:42:03.5733152Z 2022-09-27T15:42:03.5733247Z Generating XML reports... 2022-09-27T15:42:03.5770419Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154201.xml 2022-09-27T15:42:04.9508840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:04.9509205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:04.9517424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:04.9518019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:05.1096984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvffqkfh 2022-09-27T15:42:05.1098308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvffqkfh/_remote_module_non_scriptable.py 2022-09-27T15:42:05.3697736Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:05.3707357Z 2022-09-27T15:42:05.3707585Z Running tests... 2022-09-27T15:42:05.3708182Z ---------------------------------------------------------------------- 2022-09-27T15:42:05.6135652Z test_async_class_rref_proxy_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4779 2022-09-27T15:42:05.6163685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4780 2022-09-27T15:42:05.6193178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4781 2022-09-27T15:42:05.6223339Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4782 2022-09-27T15:42:06.7368578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:06.7369226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:06.7369974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:06.7370490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:06.7677776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:06.7678454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:06.7745874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:06.7746404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:06.7747067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:06.7747561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:06.7974514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:06.7975130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:06.8064230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:06.8064836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:06.8342144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:06.8342783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:06.9026027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4iffbtw 2022-09-27T15:42:06.9026806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4iffbtw/_remote_module_non_scriptable.py 2022-09-27T15:42:06.9027537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskmjtabp 2022-09-27T15:42:06.9028393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskmjtabp/_remote_module_non_scriptable.py 2022-09-27T15:42:06.9323208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfe14k1w 2022-09-27T15:42:06.9324017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfe14k1w/_remote_module_non_scriptable.py 2022-09-27T15:42:06.9604273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_9a3tsxk 2022-09-27T15:42:06.9605212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_9a3tsxk/_remote_module_non_scriptable.py 2022-09-27T15:42:07.1598013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:42:07.1599386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:42:07.1920731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:42:07.2390636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:42:07.9281766Z ok (2.557s) 2022-09-27T15:42:07.9282110Z 2022-09-27T15:42:07.9282575Z ---------------------------------------------------------------------- 2022-09-27T15:42:07.9282826Z Ran 1 test in 2.557s 2022-09-27T15:42:07.9282943Z 2022-09-27T15:42:07.9283003Z OK 2022-09-27T15:42:07.9283094Z 2022-09-27T15:42:07.9283177Z Generating XML reports... 2022-09-27T15:42:07.9318301Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154205.xml 2022-09-27T15:42:09.2859277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:09.2859650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:09.2868127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:09.2868488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:09.4459373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09x9i_4q 2022-09-27T15:42:09.4460561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09x9i_4q/_remote_module_non_scriptable.py 2022-09-27T15:42:09.7104432Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:09.7114491Z 2022-09-27T15:42:09.7114759Z Running tests... 2022-09-27T15:42:09.7115171Z ---------------------------------------------------------------------- 2022-09-27T15:42:09.9534809Z test_async_function_chained (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5002 2022-09-27T15:42:09.9562384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5003 2022-09-27T15:42:09.9590726Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5004 2022-09-27T15:42:09.9620479Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5005 2022-09-27T15:42:11.1036381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:11.1037029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:11.1096991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:11.1097659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:11.1177892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:11.1178417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:11.1300098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:11.1300771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:11.1426953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:11.1427486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:11.1485481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:11.1486028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:11.1565849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:11.1566684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:11.1686042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:11.1686930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:11.2696743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzihcq1iy 2022-09-27T15:42:11.2697901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzihcq1iy/_remote_module_non_scriptable.py 2022-09-27T15:42:11.2749072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx39qee9l 2022-09-27T15:42:11.2750243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx39qee9l/_remote_module_non_scriptable.py 2022-09-27T15:42:11.3014725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbo752j9 2022-09-27T15:42:11.3015464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_ljp4kd 2022-09-27T15:42:11.3016114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbo752j9/_remote_module_non_scriptable.py 2022-09-27T15:42:11.3016805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_ljp4kd/_remote_module_non_scriptable.py 2022-09-27T15:42:11.5296870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:42:11.5345970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:42:11.5622986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:42:11.5623797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:42:12.1677523Z ok (2.456s) 2022-09-27T15:42:12.1677797Z 2022-09-27T15:42:12.1678284Z ---------------------------------------------------------------------- 2022-09-27T15:42:12.1678582Z Ran 1 test in 2.456s 2022-09-27T15:42:12.1678699Z 2022-09-27T15:42:12.1678747Z OK 2022-09-27T15:42:12.1678838Z 2022-09-27T15:42:12.1678930Z Generating XML reports... 2022-09-27T15:42:12.1715470Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154209.xml 2022-09-27T15:42:13.5545240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:13.5545625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:13.5553845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:13.5554214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:13.7116701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzi0roafy 2022-09-27T15:42:13.7117266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzi0roafy/_remote_module_non_scriptable.py 2022-09-27T15:42:13.9735439Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:13.9744936Z 2022-09-27T15:42:13.9745010Z Running tests... 2022-09-27T15:42:13.9745576Z ---------------------------------------------------------------------- 2022-09-27T15:42:14.2155939Z test_async_function_chained_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5225 2022-09-27T15:42:14.2181939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5226 2022-09-27T15:42:14.2210201Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5227 2022-09-27T15:42:14.2240069Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5228 2022-09-27T15:42:15.3122530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:15.3123477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:15.3133924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:15.3134834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:15.3218918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:15.3219430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:15.3229417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:15.3230054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:15.3515504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:15.3516172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:15.3530644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:15.3531294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:15.3614174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:15.3614801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:15.3663760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:15.3664391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:15.4814809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_phj_0c1 2022-09-27T15:42:15.4815553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_phj_0c1/_remote_module_non_scriptable.py 2022-09-27T15:42:15.4828725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxl5px9o 2022-09-27T15:42:15.4830343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxl5px9o/_remote_module_non_scriptable.py 2022-09-27T15:42:15.4903506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu1x1xq9q 2022-09-27T15:42:15.4905196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu1x1xq9q/_remote_module_non_scriptable.py 2022-09-27T15:42:15.5036918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpg_jwtm 2022-09-27T15:42:15.5038296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpg_jwtm/_remote_module_non_scriptable.py 2022-09-27T15:42:15.7377546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:42:15.7418712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:42:15.7464659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:42:15.7601083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:42:16.4295695Z ok (2.455s) 2022-09-27T15:42:16.4295959Z 2022-09-27T15:42:16.4296343Z ---------------------------------------------------------------------- 2022-09-27T15:42:16.4296593Z Ran 1 test in 2.455s 2022-09-27T15:42:16.4296711Z 2022-09-27T15:42:16.4296788Z OK 2022-09-27T15:42:16.4296914Z 2022-09-27T15:42:16.4297011Z Generating XML reports... 2022-09-27T15:42:16.4330622Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154213.xml 2022-09-27T15:42:17.7990858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:17.7991326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:17.7999729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:17.8000096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:17.9533455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fccbrlm 2022-09-27T15:42:17.9561060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fccbrlm/_remote_module_non_scriptable.py 2022-09-27T15:42:18.2143973Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:18.2154263Z 2022-09-27T15:42:18.2154776Z Running tests... 2022-09-27T15:42:18.2155399Z ---------------------------------------------------------------------- 2022-09-27T15:42:18.4553035Z test_async_function_multi_chained (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5448 2022-09-27T15:42:18.4585417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5449 2022-09-27T15:42:18.4619607Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5450 2022-09-27T15:42:18.4649919Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5451 2022-09-27T15:42:19.5803835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:19.5804464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:19.6075559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:19.6076233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:19.6177908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:19.6178444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:19.6200560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:19.6201056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:19.6301795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:19.6302447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:19.6456336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:19.6457071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:19.6593029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:19.6593672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:19.6698003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:19.6698669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:19.7466735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5p5zfweg 2022-09-27T15:42:19.7467657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5p5zfweg/_remote_module_non_scriptable.py 2022-09-27T15:42:19.7739453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpibs_oprh 2022-09-27T15:42:19.7740664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpibs_oprh/_remote_module_non_scriptable.py 2022-09-27T15:42:19.7860879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojzsnqv4 2022-09-27T15:42:19.7861673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojzsnqv4/_remote_module_non_scriptable.py 2022-09-27T15:42:19.7978778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkcjrxhi4 2022-09-27T15:42:19.7979731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkcjrxhi4/_remote_module_non_scriptable.py 2022-09-27T15:42:20.0081250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:42:20.0357452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:42:20.0464601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:42:20.0586960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:42:20.6704710Z ok (2.455s) 2022-09-27T15:42:20.6704961Z 2022-09-27T15:42:20.6705470Z ---------------------------------------------------------------------- 2022-09-27T15:42:20.6705722Z Ran 1 test in 2.455s 2022-09-27T15:42:20.6705850Z 2022-09-27T15:42:20.6705898Z OK 2022-09-27T15:42:20.6705990Z 2022-09-27T15:42:20.6706084Z Generating XML reports... 2022-09-27T15:42:20.6741953Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154218.xml 2022-09-27T15:42:22.0451897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:22.0452291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:22.0460766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:22.0461186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:22.2069403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpibp87_17 2022-09-27T15:42:22.2070591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpibp87_17/_remote_module_non_scriptable.py 2022-09-27T15:42:22.4708529Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:22.4718911Z 2022-09-27T15:42:22.4719057Z Running tests... 2022-09-27T15:42:22.4719482Z ---------------------------------------------------------------------- 2022-09-27T15:42:22.7229321Z test_async_function_multi_chained_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5671 2022-09-27T15:42:22.7256610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5672 2022-09-27T15:42:22.7285214Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5673 2022-09-27T15:42:22.7315179Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5674 2022-09-27T15:42:23.8769511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:23.8770211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:23.8850766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:23.8851385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:23.8965736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:23.8966494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:23.9023235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:23.9023904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:23.9167375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:23.9167933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:23.9239406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:23.9239938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:23.9353937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:23.9354926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:23.9400136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:23.9400765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:24.0428288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphto9o3tv 2022-09-27T15:42:24.0429446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphto9o3tv/_remote_module_non_scriptable.py 2022-09-27T15:42:24.0500005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzdjcjkg 2022-09-27T15:42:24.0501640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzdjcjkg/_remote_module_non_scriptable.py 2022-09-27T15:42:24.0618740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp15rca54 2022-09-27T15:42:24.0620118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp15rca54/_remote_module_non_scriptable.py 2022-09-27T15:42:24.0651490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1zl9zk_ 2022-09-27T15:42:24.0653044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1zl9zk_/_remote_module_non_scriptable.py 2022-09-27T15:42:24.2999833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:42:24.3071152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:42:24.3203487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:42:24.3228837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:42:24.9370909Z ok (2.465s) 2022-09-27T15:42:24.9371166Z 2022-09-27T15:42:24.9371655Z ---------------------------------------------------------------------- 2022-09-27T15:42:24.9371913Z Ran 1 test in 2.465s 2022-09-27T15:42:24.9372048Z 2022-09-27T15:42:24.9372095Z OK 2022-09-27T15:42:24.9372191Z 2022-09-27T15:42:24.9372288Z Generating XML reports... 2022-09-27T15:42:24.9405755Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154222.xml 2022-09-27T15:42:26.2995723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:26.2996106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:26.3004325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:26.3004767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:26.4615587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtdml5x0 2022-09-27T15:42:26.4616650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtdml5x0/_remote_module_non_scriptable.py 2022-09-27T15:42:26.7240131Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:26.7249984Z 2022-09-27T15:42:26.7250127Z Running tests... 2022-09-27T15:42:26.7250508Z ---------------------------------------------------------------------- 2022-09-27T15:42:26.9763691Z test_async_function_multi_chained_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5894 2022-09-27T15:42:26.9791726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5895 2022-09-27T15:42:26.9819981Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5896 2022-09-27T15:42:26.9850799Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5897 2022-09-27T15:42:28.0756314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:28.0757064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:28.0973213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:28.0973866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:28.1124042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:28.1124616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:28.1339088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:28.1339714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:28.1353218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:28.1353847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:28.1664789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:28.1665459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:28.1727048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:28.1727584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:28.2034047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:28.2034876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:28.2396949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_veb0tk 2022-09-27T15:42:28.2398635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_veb0tk/_remote_module_non_scriptable.py 2022-09-27T15:42:28.2610993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppc6pn1xp 2022-09-27T15:42:28.2611686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppc6pn1xp/_remote_module_non_scriptable.py 2022-09-27T15:42:28.3005937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiotx70eg 2022-09-27T15:42:28.3007434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiotx70eg/_remote_module_non_scriptable.py 2022-09-27T15:42:28.3300032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwrz10t9 2022-09-27T15:42:28.3300784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwrz10t9/_remote_module_non_scriptable.py 2022-09-27T15:42:28.4994356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:42:28.5190479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:42:28.5593849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:42:28.5863916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:42:29.1905518Z ok (2.465s) 2022-09-27T15:42:29.1905748Z 2022-09-27T15:42:29.1906044Z ---------------------------------------------------------------------- 2022-09-27T15:42:29.1906292Z Ran 1 test in 2.465s 2022-09-27T15:42:29.1906409Z 2022-09-27T15:42:29.1906458Z OK 2022-09-27T15:42:29.1906550Z 2022-09-27T15:42:29.1906648Z Generating XML reports... 2022-09-27T15:42:29.1942230Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154226.xml 2022-09-27T15:42:30.5655029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:30.5655750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:30.5664672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:30.5665271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:30.7271177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgcxa2iwd 2022-09-27T15:42:30.7272189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgcxa2iwd/_remote_module_non_scriptable.py 2022-09-27T15:42:30.9874404Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:30.9884244Z 2022-09-27T15:42:30.9884434Z Running tests... 2022-09-27T15:42:30.9884767Z ---------------------------------------------------------------------- 2022-09-27T15:42:31.2334572Z test_async_function_multi_fanout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6117 2022-09-27T15:42:31.2361057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6118 2022-09-27T15:42:31.2388764Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6119 2022-09-27T15:42:31.2418559Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6120 2022-09-27T15:42:32.4066184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:32.4066798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:32.4351813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:32.4352437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:32.4462905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:32.4463555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:32.4538480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:32.4539121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:32.4743034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:32.4743684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:32.4907332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:32.4907994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:32.5299788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:32.5300434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:32.5675993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:32.5676654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:32.5738138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnexgu6hq 2022-09-27T15:42:32.5738969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnexgu6hq/_remote_module_non_scriptable.py 2022-09-27T15:42:32.6028505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21q2yhk1 2022-09-27T15:42:32.6029293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21q2yhk1/_remote_module_non_scriptable.py 2022-09-27T15:42:32.6184862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ws1wstd 2022-09-27T15:42:32.6185671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ws1wstd/_remote_module_non_scriptable.py 2022-09-27T15:42:32.6966272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpau9cfok0 2022-09-27T15:42:32.6967143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpau9cfok0/_remote_module_non_scriptable.py 2022-09-27T15:42:32.8339749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:42:32.8645426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:42:32.8767541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:42:32.9560931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:42:33.5474049Z ok (2.559s) 2022-09-27T15:42:33.5474276Z 2022-09-27T15:42:33.5474835Z ---------------------------------------------------------------------- 2022-09-27T15:42:33.5475256Z Ran 1 test in 2.559s 2022-09-27T15:42:33.5475463Z 2022-09-27T15:42:33.5475569Z OK 2022-09-27T15:42:33.5475739Z 2022-09-27T15:42:33.5476199Z Generating XML reports... 2022-09-27T15:42:33.5510794Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154230.xml 2022-09-27T15:42:34.9329648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:34.9330040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:34.9337262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:34.9337671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:35.0890074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6zw3x6nh 2022-09-27T15:42:35.0890923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6zw3x6nh/_remote_module_non_scriptable.py 2022-09-27T15:42:35.3493212Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:35.3502668Z 2022-09-27T15:42:35.3503076Z Running tests... 2022-09-27T15:42:35.3503470Z ---------------------------------------------------------------------- 2022-09-27T15:42:35.5950014Z test_async_function_multi_fanout_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6340 2022-09-27T15:42:35.5977307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6341 2022-09-27T15:42:35.6004995Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6342 2022-09-27T15:42:35.6034635Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6343 2022-09-27T15:42:36.7365083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:36.7365676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:36.7394975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:36.7395660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:36.7413713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:36.7414347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:36.7511581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:36.7512096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:36.7753082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:36.7753610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:36.7786893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:36.7787559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:36.7805783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:36.7806422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:36.7879731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:36.7880393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:36.9014781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6_s5mm6 2022-09-27T15:42:36.9015887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6_s5mm6/_remote_module_non_scriptable.py 2022-09-27T15:42:36.9051786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9owjda4 2022-09-27T15:42:36.9052844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9owjda4/_remote_module_non_scriptable.py 2022-09-27T15:42:36.9073495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5fl80teb 2022-09-27T15:42:36.9075692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5fl80teb/_remote_module_non_scriptable.py 2022-09-27T15:42:36.9132849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhgx6i_d 2022-09-27T15:42:36.9134021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhgx6i_d/_remote_module_non_scriptable.py 2022-09-27T15:42:37.1601181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:42:37.1629989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:42:37.1713834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:42:37.1852446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:42:37.8089797Z ok (2.458s) 2022-09-27T15:42:37.8090072Z 2022-09-27T15:42:37.8090497Z ---------------------------------------------------------------------- 2022-09-27T15:42:37.8090758Z Ran 1 test in 2.459s 2022-09-27T15:42:37.8090875Z 2022-09-27T15:42:37.8090938Z OK 2022-09-27T15:42:37.8091031Z 2022-09-27T15:42:37.8091155Z Generating XML reports... 2022-09-27T15:42:37.8126400Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154235.xml 2022-09-27T15:42:39.1660233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:39.1660606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:39.1668589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:39.1669128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:39.3229829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptkh99dqv 2022-09-27T15:42:39.3230723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptkh99dqv/_remote_module_non_scriptable.py 2022-09-27T15:42:39.5841353Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:39.5850892Z 2022-09-27T15:42:39.5851020Z Running tests... 2022-09-27T15:42:39.5851600Z ---------------------------------------------------------------------- 2022-09-27T15:42:39.8312711Z test_async_function_multi_fanout_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6563 2022-09-27T15:42:39.8339982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6564 2022-09-27T15:42:39.8367942Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6565 2022-09-27T15:42:39.8398454Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6566 2022-09-27T15:42:40.9695140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:40.9695804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:40.9960773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:40.9961488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:41.0049384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:41.0049932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:41.0062609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:41.0063240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:41.0325784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:41.0326473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:41.0337281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:41.0337965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:41.0426403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:41.0426950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:41.0689530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:41.0690224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:41.1307788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7mf7hv9 2022-09-27T15:42:41.1308844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7mf7hv9/_remote_module_non_scriptable.py 2022-09-27T15:42:41.1590380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprp4hvt5k 2022-09-27T15:42:41.1591503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprp4hvt5k/_remote_module_non_scriptable.py 2022-09-27T15:42:41.1686520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqn4q0wgz 2022-09-27T15:42:41.1688533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqn4q0wgz/_remote_module_non_scriptable.py 2022-09-27T15:42:41.1942248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7g5463e 2022-09-27T15:42:41.1943524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7g5463e/_remote_module_non_scriptable.py 2022-09-27T15:42:41.3895635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:42:41.4196117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:42:41.4496852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:42:41.4548450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:42:42.0453954Z ok (2.460s) 2022-09-27T15:42:42.0454217Z 2022-09-27T15:42:42.0454630Z ---------------------------------------------------------------------- 2022-09-27T15:42:42.0454990Z Ran 1 test in 2.460s 2022-09-27T15:42:42.0455186Z 2022-09-27T15:42:42.0455284Z OK 2022-09-27T15:42:42.0455433Z 2022-09-27T15:42:42.0455575Z Generating XML reports... 2022-09-27T15:42:42.0491908Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154239.xml 2022-09-27T15:42:43.4023221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:43.4023699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:43.4035346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:43.4035724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:43.5693338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxw7mqyus 2022-09-27T15:42:43.5693989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxw7mqyus/_remote_module_non_scriptable.py 2022-09-27T15:42:43.8369780Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:43.8379515Z 2022-09-27T15:42:43.8379647Z Running tests... 2022-09-27T15:42:43.8380310Z ---------------------------------------------------------------------- 2022-09-27T15:42:44.0818294Z test_async_function_nested (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6786 2022-09-27T15:42:44.0845404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6787 2022-09-27T15:42:44.0873820Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6788 2022-09-27T15:42:44.0902966Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6789 2022-09-27T15:42:45.2151892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:45.2152540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:45.2312126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:45.2312821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:45.2541646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:45.2542310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:45.2687753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:45.2688425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:45.2720964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:45.2721612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:45.2722373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:45.2722991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:45.3096266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:45.3097093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:45.3097917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:45.3098495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:45.3846016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatf55vla 2022-09-27T15:42:45.3846799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatf55vla/_remote_module_non_scriptable.py 2022-09-27T15:42:45.3960296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkod1l53h 2022-09-27T15:42:45.3961213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkod1l53h/_remote_module_non_scriptable.py 2022-09-27T15:42:45.4375065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1ujnwwx 2022-09-27T15:42:45.4376083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1ujnwwx/_remote_module_non_scriptable.py 2022-09-27T15:42:45.4532901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4r288u9 2022-09-27T15:42:45.4533717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4r288u9/_remote_module_non_scriptable.py 2022-09-27T15:42:45.6443759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:42:45.6529197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:42:45.6969009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:42:45.7112267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:42:46.3961487Z ok (2.558s) 2022-09-27T15:42:46.3961785Z 2022-09-27T15:42:46.3962198Z ---------------------------------------------------------------------- 2022-09-27T15:42:46.3962459Z Ran 1 test in 2.558s 2022-09-27T15:42:46.3962576Z 2022-09-27T15:42:46.3962623Z OK 2022-09-27T15:42:46.3962724Z 2022-09-27T15:42:46.3962817Z Generating XML reports... 2022-09-27T15:42:46.3996900Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154243.xml 2022-09-27T15:42:47.7557904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:47.7558567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:47.7567262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:47.7567935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:47.9112278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppbykekhe 2022-09-27T15:42:47.9112942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppbykekhe/_remote_module_non_scriptable.py 2022-09-27T15:42:48.1716085Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:48.1726807Z 2022-09-27T15:42:48.1727141Z Running tests... 2022-09-27T15:42:48.1727769Z ---------------------------------------------------------------------- 2022-09-27T15:42:48.4151617Z test_async_function_nested_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7009 2022-09-27T15:42:48.4179711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7010 2022-09-27T15:42:48.4209458Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7011 2022-09-27T15:42:48.4240945Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7012 2022-09-27T15:42:49.5229569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:49.5230191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:49.5292105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:49.5292627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:49.5371082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:49.5371576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:49.5556023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:49.5556525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:49.5618834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:49.5619672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:49.5692737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:49.5693764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:49.5766625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:49.5767233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:49.5944568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:49.5945165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:49.6898150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnp73tun5 2022-09-27T15:42:49.6898951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnp73tun5/_remote_module_non_scriptable.py 2022-09-27T15:42:49.6977713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77zy1wue 2022-09-27T15:42:49.6978724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77zy1wue/_remote_module_non_scriptable.py 2022-09-27T15:42:49.7040758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpmayz0z 2022-09-27T15:42:49.7042126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpmayz0z/_remote_module_non_scriptable.py 2022-09-27T15:42:49.7206314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43a0jssm 2022-09-27T15:42:49.7207062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43a0jssm/_remote_module_non_scriptable.py 2022-09-27T15:42:49.9452823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:42:49.9552238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:42:49.9590613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:42:49.9770536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:42:50.7297026Z ok (2.557s) 2022-09-27T15:42:50.7297286Z 2022-09-27T15:42:50.7297803Z ---------------------------------------------------------------------- 2022-09-27T15:42:50.7298200Z Ran 1 test in 2.557s 2022-09-27T15:42:50.7298321Z 2022-09-27T15:42:50.7298381Z OK 2022-09-27T15:42:50.7298475Z 2022-09-27T15:42:50.7298571Z Generating XML reports... 2022-09-27T15:42:50.7334259Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154248.xml 2022-09-27T15:42:52.0954211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:52.0954760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:52.0962142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:52.0962515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:52.2551074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ysa9d8d 2022-09-27T15:42:52.2552071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ysa9d8d/_remote_module_non_scriptable.py 2022-09-27T15:42:52.5208295Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:52.5217782Z 2022-09-27T15:42:52.5217932Z Running tests... 2022-09-27T15:42:52.5218730Z ---------------------------------------------------------------------- 2022-09-27T15:42:52.7645058Z test_async_function_raise (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7232 2022-09-27T15:42:52.7672151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7233 2022-09-27T15:42:52.7699717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7234 2022-09-27T15:42:52.7728977Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7235 2022-09-27T15:42:53.8670645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:53.8671422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:53.8783028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:53.8783595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:53.8926689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:53.8927218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:53.8991556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:53.8992299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:53.9060444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:53.9061014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:53.9171610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:53.9172298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:53.9305177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:53.9305847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:53.9360283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:53.9360972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:54.0326236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbh406u_ 2022-09-27T15:42:54.0327183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbh406u_/_remote_module_non_scriptable.py 2022-09-27T15:42:54.0452925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52j_r0th 2022-09-27T15:42:54.0453826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52j_r0th/_remote_module_non_scriptable.py 2022-09-27T15:42:54.0562837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxf5fb_tk 2022-09-27T15:42:54.0563760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxf5fb_tk/_remote_module_non_scriptable.py 2022-09-27T15:42:54.0625776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpen3qydli 2022-09-27T15:42:54.0627134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpen3qydli/_remote_module_non_scriptable.py 2022-09-27T15:42:54.2901440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:42:54.3089762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:42:54.3144143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:42:54.3237847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:42:54.5430618Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:42:54.5431431Z RuntimeError('Expected error') 2022-09-27T15:42:54.5431907Z Traceback (most recent call last): 2022-09-27T15:42:54.5434803Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:42:54.5435694Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:42:54.5436990Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-09-27T15:42:54.5437708Z return fn(*args, **kwargs) 2022-09-27T15:42:54.5438610Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-09-27T15:42:54.5439285Z raise RuntimeError("Expected error") 2022-09-27T15:42:54.5439725Z RuntimeError: Expected error 2022-09-27T15:42:54.5439993Z 2022-09-27T15:42:54.5588425Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:42:54.5589336Z RuntimeError('Expected error') 2022-09-27T15:42:54.5589956Z Traceback (most recent call last): 2022-09-27T15:42:54.5590978Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:42:54.5591949Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:42:54.5593196Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-09-27T15:42:54.5593939Z return fn(*args, **kwargs) 2022-09-27T15:42:54.5595159Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-09-27T15:42:54.5595849Z raise RuntimeError("Expected error") 2022-09-27T15:42:54.5596313Z RuntimeError: Expected error 2022-09-27T15:42:54.5596553Z 2022-09-27T15:42:54.5626666Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:42:54.5627468Z RuntimeError('Expected error') 2022-09-27T15:42:54.5627853Z Traceback (most recent call last): 2022-09-27T15:42:54.5628531Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:42:54.5629098Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:42:54.5629789Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-09-27T15:42:54.5630279Z return fn(*args, **kwargs) 2022-09-27T15:42:54.5631066Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-09-27T15:42:54.5631712Z raise RuntimeError("Expected error") 2022-09-27T15:42:54.5632093Z RuntimeError: Expected error 2022-09-27T15:42:54.5632307Z 2022-09-27T15:42:54.5689227Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:42:54.5689936Z RuntimeError('Expected error') 2022-09-27T15:42:54.5690398Z Traceback (most recent call last): 2022-09-27T15:42:54.5691274Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:42:54.5691977Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:42:54.5692873Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-09-27T15:42:54.5693473Z return fn(*args, **kwargs) 2022-09-27T15:42:54.5694366Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-09-27T15:42:54.5695041Z raise RuntimeError("Expected error") 2022-09-27T15:42:54.5695516Z RuntimeError: Expected error 2022-09-27T15:42:54.5695786Z 2022-09-27T15:42:54.9785735Z ok (2.456s) 2022-09-27T15:42:54.9785905Z 2022-09-27T15:42:54.9786215Z ---------------------------------------------------------------------- 2022-09-27T15:42:54.9786466Z Ran 1 test in 2.457s 2022-09-27T15:42:54.9786584Z 2022-09-27T15:42:54.9786632Z OK 2022-09-27T15:42:54.9786724Z 2022-09-27T15:42:54.9786816Z Generating XML reports... 2022-09-27T15:42:54.9822178Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154252.xml 2022-09-27T15:42:56.3543955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:56.3544336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:56.3552962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:56.3553381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:56.5125346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxsybmza 2022-09-27T15:42:56.5125810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxsybmza/_remote_module_non_scriptable.py 2022-09-27T15:42:56.7774291Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:56.7784286Z 2022-09-27T15:42:56.7784691Z Running tests... 2022-09-27T15:42:56.7785086Z ---------------------------------------------------------------------- 2022-09-27T15:42:57.0102313Z test_async_function_raise_async (__main__.TensorPipeRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76907 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.231s) 2022-09-27T15:42:57.0102820Z 2022-09-27T15:42:57.0103044Z ---------------------------------------------------------------------- 2022-09-27T15:42:57.0103287Z Ran 1 test in 0.232s 2022-09-27T15:42:57.0103402Z 2022-09-27T15:42:57.0103473Z OK (skipped=1) 2022-09-27T15:42:57.0103580Z 2022-09-27T15:42:57.0103662Z Generating XML reports... 2022-09-27T15:42:57.0126943Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154256.xml 2022-09-27T15:42:58.3190687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:42:58.3191063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:42:58.3198778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:42:58.3199259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:42:58.4792734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmb55dmg 2022-09-27T15:42:58.4793822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmb55dmg/_remote_module_non_scriptable.py 2022-09-27T15:42:58.7405263Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:42:58.7415408Z 2022-09-27T15:42:58.7415908Z Running tests... 2022-09-27T15:42:58.7416301Z ---------------------------------------------------------------------- 2022-09-27T15:42:58.9855613Z test_async_function_raise_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7465 2022-09-27T15:42:58.9883171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7466 2022-09-27T15:42:58.9911864Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7467 2022-09-27T15:42:58.9941288Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7468 2022-09-27T15:43:00.1220034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:00.1220671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:00.1336999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:00.1337538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:00.1425782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:00.1426847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:00.1611015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:00.1611675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:00.1658532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:00.1659516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:00.1728450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:00.1728983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:00.1807175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:00.1807742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:00.2026936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:00.2027579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:00.2891041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92ax8on9 2022-09-27T15:43:00.2893050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92ax8on9/_remote_module_non_scriptable.py 2022-09-27T15:43:00.3010383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmcf9qx_ 2022-09-27T15:43:00.3011799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmcf9qx_/_remote_module_non_scriptable.py 2022-09-27T15:43:00.3071664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhnp9vbp 2022-09-27T15:43:00.3073052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhnp9vbp/_remote_module_non_scriptable.py 2022-09-27T15:43:00.3293612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogq93fm4 2022-09-27T15:43:00.3294820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogq93fm4/_remote_module_non_scriptable.py 2022-09-27T15:43:00.5452847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:00.5601574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:00.5611873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:00.5870834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:00.7993190Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:43:00.7993978Z RuntimeError('Expected error') 2022-09-27T15:43:00.7994437Z Traceback (most recent call last): 2022-09-27T15:43:00.7995513Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:43:00.7996210Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:43:00.7997701Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-09-27T15:43:00.7998287Z return fn(*args, **kwargs) 2022-09-27T15:43:00.7999174Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-09-27T15:43:00.7999850Z raise RuntimeError("Expected error") 2022-09-27T15:43:00.8000303Z RuntimeError: Expected error 2022-09-27T15:43:00.8000575Z 2022-09-27T15:43:00.8191932Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:43:00.8192542Z RuntimeError('Expected error') 2022-09-27T15:43:00.8192936Z Traceback (most recent call last): 2022-09-27T15:43:00.8193604Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:43:00.8194175Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:43:00.8195226Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-09-27T15:43:00.8196204Z return fn(*args, **kwargs) 2022-09-27T15:43:00.8196923Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-09-27T15:43:00.8197589Z raise RuntimeError("Expected error") 2022-09-27T15:43:00.8197978Z RuntimeError: Expected error 2022-09-27T15:43:00.8198199Z 2022-09-27T15:43:00.8289617Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:43:00.8290398Z RuntimeError('Expected error') 2022-09-27T15:43:00.8290872Z Traceback (most recent call last): 2022-09-27T15:43:00.8291747Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:43:00.8292448Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:43:00.8293363Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-09-27T15:43:00.8293973Z return fn(*args, **kwargs) 2022-09-27T15:43:00.8294881Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-09-27T15:43:00.8295554Z raise RuntimeError("Expected error") 2022-09-27T15:43:00.8296021Z RuntimeError: Expected error 2022-09-27T15:43:00.8296298Z 2022-09-27T15:43:00.8330079Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:43:00.8330735Z RuntimeError('Expected error') 2022-09-27T15:43:00.8331199Z Traceback (most recent call last): 2022-09-27T15:43:00.8333963Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:43:00.8337409Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:43:00.8338592Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-09-27T15:43:00.8341786Z return fn(*args, **kwargs) 2022-09-27T15:43:00.8345293Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-09-27T15:43:00.8348555Z raise RuntimeError("Expected error") 2022-09-27T15:43:00.8349394Z RuntimeError: Expected error 2022-09-27T15:43:00.8349715Z 2022-09-27T15:43:01.1997377Z ok (2.458s) 2022-09-27T15:43:01.1997627Z 2022-09-27T15:43:01.1998117Z ---------------------------------------------------------------------- 2022-09-27T15:43:01.1998449Z Ran 1 test in 2.458s 2022-09-27T15:43:01.1998567Z 2022-09-27T15:43:01.1998627Z OK 2022-09-27T15:43:01.1998718Z 2022-09-27T15:43:01.1998815Z Generating XML reports... 2022-09-27T15:43:01.2034079Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154258.xml 2022-09-27T15:43:02.6325561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:02.6325929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:02.6334399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:02.6334766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:02.7942215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsu5qwny7 2022-09-27T15:43:02.7943805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsu5qwny7/_remote_module_non_scriptable.py 2022-09-27T15:43:03.0564572Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:43:03.0572993Z 2022-09-27T15:43:03.0573232Z Running tests... 2022-09-27T15:43:03.0573976Z ---------------------------------------------------------------------- 2022-09-27T15:43:03.3070681Z test_async_function_simple (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7688 2022-09-27T15:43:03.3097644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7689 2022-09-27T15:43:03.3126097Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7690 2022-09-27T15:43:03.3156304Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7691 2022-09-27T15:43:04.4469614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:04.4470244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:04.4774286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:04.4774948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:04.4822086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:04.4822765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:04.5047023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:04.5145511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:04.5146320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:04.5146841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:04.5197226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:04.5197793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:04.5423285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:04.5423842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:04.5562262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:04.5562935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:04.6100615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzuz8ltc5 2022-09-27T15:43:04.6101436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzuz8ltc5/_remote_module_non_scriptable.py 2022-09-27T15:43:04.6409527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ypcwxxv 2022-09-27T15:43:04.6410320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ypcwxxv/_remote_module_non_scriptable.py 2022-09-27T15:43:04.6687553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gjc7d_n 2022-09-27T15:43:04.6688452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gjc7d_n/_remote_module_non_scriptable.py 2022-09-27T15:43:04.6812418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0uta4q8 2022-09-27T15:43:04.6813548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0uta4q8/_remote_module_non_scriptable.py 2022-09-27T15:43:04.8676988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:04.8988003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:04.9247896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:04.9367731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:05.6214769Z ok (2.564s) 2022-09-27T15:43:05.6215031Z 2022-09-27T15:43:05.6215545Z ---------------------------------------------------------------------- 2022-09-27T15:43:05.6215812Z Ran 1 test in 2.564s 2022-09-27T15:43:05.6215930Z 2022-09-27T15:43:05.6215994Z OK 2022-09-27T15:43:05.6216088Z 2022-09-27T15:43:05.6216182Z Generating XML reports... 2022-09-27T15:43:05.6251419Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154303.xml 2022-09-27T15:43:07.0115171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:07.0115865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:07.0123819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:07.0124187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:07.1820949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadcoy_7h 2022-09-27T15:43:07.1821919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadcoy_7h/_remote_module_non_scriptable.py 2022-09-27T15:43:07.4488248Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:43:07.4498135Z 2022-09-27T15:43:07.4498341Z Running tests... 2022-09-27T15:43:07.4498701Z ---------------------------------------------------------------------- 2022-09-27T15:43:07.7007133Z test_async_function_with_future_ctor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7911 2022-09-27T15:43:07.7037579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7912 2022-09-27T15:43:07.7066444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7913 2022-09-27T15:43:07.7097715Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7914 2022-09-27T15:43:08.8404201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:08.8404892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:08.8474162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:08.8474880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:08.8790129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:08.8790743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:08.8854128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:08.8854789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:08.8861021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:08.8861782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:08.9082617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:08.9083274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:08.9238588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:08.9239247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:08.9448455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:08.9449117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:09.0071633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmk6_3xuq 2022-09-27T15:43:09.0072453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmk6_3xuq/_remote_module_non_scriptable.py 2022-09-27T15:43:09.0129683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_zpj0osp 2022-09-27T15:43:09.0131124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_zpj0osp/_remote_module_non_scriptable.py 2022-09-27T15:43:09.0494172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplfakcdl6 2022-09-27T15:43:09.0494891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplfakcdl6/_remote_module_non_scriptable.py 2022-09-27T15:43:09.0710930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavrjerj2 2022-09-27T15:43:09.0712018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavrjerj2/_remote_module_non_scriptable.py 2022-09-27T15:43:09.2667881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:09.2886379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:09.3074312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:09.3293136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:10.0155627Z ok (2.565s) 2022-09-27T15:43:10.0155797Z 2022-09-27T15:43:10.0156184Z ---------------------------------------------------------------------- 2022-09-27T15:43:10.0156493Z Ran 1 test in 2.566s 2022-09-27T15:43:10.0156601Z 2022-09-27T15:43:10.0156662Z OK 2022-09-27T15:43:10.0156752Z 2022-09-27T15:43:10.0156844Z Generating XML reports... 2022-09-27T15:43:10.0191871Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154307.xml 2022-09-27T15:43:11.4047467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:11.4047866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:11.4056726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:11.4057093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:11.5700795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadkn52j5 2022-09-27T15:43:11.5701694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadkn52j5/_remote_module_non_scriptable.py 2022-09-27T15:43:11.8310550Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:43:11.8320156Z 2022-09-27T15:43:11.8320277Z Running tests... 2022-09-27T15:43:11.8320709Z ---------------------------------------------------------------------- 2022-09-27T15:43:12.0822603Z test_async_function_with_future_ctor_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8134 2022-09-27T15:43:12.0849646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8135 2022-09-27T15:43:12.0877820Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8136 2022-09-27T15:43:12.0907133Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8137 2022-09-27T15:43:13.2013878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:13.2014567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:13.2382209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:13.2382974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:13.2488242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:13.2488862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:13.2539804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:13.2540454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:13.2816873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:13.2817508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:13.2875035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:13.2875696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:13.2917512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:13.2918168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:13.3204168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:13.3204792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:13.3670685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tysqtqn 2022-09-27T15:43:13.3671557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tysqtqn/_remote_module_non_scriptable.py 2022-09-27T15:43:13.4149397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxb9dkc6h 2022-09-27T15:43:13.4150217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxb9dkc6h/_remote_module_non_scriptable.py 2022-09-27T15:43:13.4186563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpttnykr_y 2022-09-27T15:43:13.4187996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpttnykr_y/_remote_module_non_scriptable.py 2022-09-27T15:43:13.4466493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwof51jnx 2022-09-27T15:43:13.4467312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwof51jnx/_remote_module_non_scriptable.py 2022-09-27T15:43:13.6239350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:13.6726850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:13.6762079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:13.7051765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:14.3964781Z ok (2.564s) 2022-09-27T15:43:14.3965073Z 2022-09-27T15:43:14.3965529Z ---------------------------------------------------------------------- 2022-09-27T15:43:14.3965770Z Ran 1 test in 2.564s 2022-09-27T15:43:14.3965888Z 2022-09-27T15:43:14.3965948Z OK 2022-09-27T15:43:14.3966040Z 2022-09-27T15:43:14.3966134Z Generating XML reports... 2022-09-27T15:43:14.4001358Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154311.xml 2022-09-27T15:43:15.7596941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:15.7597334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:15.7605612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:15.7605968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:15.9146325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaihpzjyo 2022-09-27T15:43:15.9147222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaihpzjyo/_remote_module_non_scriptable.py 2022-09-27T15:43:16.1761406Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:43:16.1771277Z 2022-09-27T15:43:16.1771390Z Running tests... 2022-09-27T15:43:16.1772081Z ---------------------------------------------------------------------- 2022-09-27T15:43:16.4288735Z test_async_function_wrong_return_type (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8357 2022-09-27T15:43:16.4316183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8358 2022-09-27T15:43:16.4344397Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8359 2022-09-27T15:43:16.4374638Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8360 2022-09-27T15:43:17.5955549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:17.5956232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:17.6337080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:17.6337753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:17.6519237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:17.6519864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:17.6520572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:17.6521113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:17.6623660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:17.6624283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:17.6901160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:17.6901756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:17.6904905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:17.6905562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:17.6999003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:17.6999616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:17.7622779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2v8smby5 2022-09-27T15:43:17.7624057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2v8smby5/_remote_module_non_scriptable.py 2022-09-27T15:43:17.8173806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0dk44v9m 2022-09-27T15:43:17.8174608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3da07qce 2022-09-27T15:43:17.8175302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0dk44v9m/_remote_module_non_scriptable.py 2022-09-27T15:43:17.8175946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3da07qce/_remote_module_non_scriptable.py 2022-09-27T15:43:17.8263180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdm8lqy0 2022-09-27T15:43:17.8264138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdm8lqy0/_remote_module_non_scriptable.py 2022-09-27T15:43:18.0194628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:18.0717098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:18.0717993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:18.0821676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:18.7433602Z ok (2.566s) 2022-09-27T15:43:18.7433872Z 2022-09-27T15:43:18.7434367Z ---------------------------------------------------------------------- 2022-09-27T15:43:18.7434824Z Ran 1 test in 2.566s 2022-09-27T15:43:18.7434944Z 2022-09-27T15:43:18.7435004Z OK 2022-09-27T15:43:18.7435097Z 2022-09-27T15:43:18.7435420Z Generating XML reports... 2022-09-27T15:43:18.7469674Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154316.xml 2022-09-27T15:43:20.1093972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:20.1094373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:20.1102668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:20.1103096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:20.2672852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpssob0fuv 2022-09-27T15:43:20.2673801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpssob0fuv/_remote_module_non_scriptable.py 2022-09-27T15:43:20.5420341Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:43:20.5429175Z 2022-09-27T15:43:20.5429321Z Running tests... 2022-09-27T15:43:20.5430297Z ---------------------------------------------------------------------- 2022-09-27T15:43:20.7866583Z test_async_function_wrong_return_type_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8580 2022-09-27T15:43:20.7895404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8581 2022-09-27T15:43:20.7924169Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8582 2022-09-27T15:43:20.7954208Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8583 2022-09-27T15:43:21.8706674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:21.8707316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:21.9081775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:21.9082447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:21.9133702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:21.9134432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:21.9247713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:21.9248384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:21.9249156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:21.9249779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:21.9521929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:21.9522606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:21.9630399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:21.9631062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:21.9637185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:21.9637808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:22.0365600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfrso380x 2022-09-27T15:43:22.0366435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfrso380x/_remote_module_non_scriptable.py 2022-09-27T15:43:22.0792635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpni8cr8v3 2022-09-27T15:43:22.0793665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpni8cr8v3/_remote_module_non_scriptable.py 2022-09-27T15:43:22.0906203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58c32afa 2022-09-27T15:43:22.0907380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58c32afa/_remote_module_non_scriptable.py 2022-09-27T15:43:22.0921104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8jkow2u 2022-09-27T15:43:22.0922450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8jkow2u/_remote_module_non_scriptable.py 2022-09-27T15:43:22.2933621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:22.3346715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:22.3464177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:22.3502140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:23.0009393Z ok (2.458s) 2022-09-27T15:43:23.0009649Z 2022-09-27T15:43:23.0010151Z ---------------------------------------------------------------------- 2022-09-27T15:43:23.0010610Z Ran 1 test in 2.458s 2022-09-27T15:43:23.0010775Z 2022-09-27T15:43:23.0010823Z OK 2022-09-27T15:43:23.0010915Z 2022-09-27T15:43:23.0011010Z Generating XML reports... 2022-09-27T15:43:23.0046070Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154320.xml 2022-09-27T15:43:24.3728303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:24.3728697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:24.3736529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:24.3736904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:24.5290365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph426s6uu 2022-09-27T15:43:24.5291269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph426s6uu/_remote_module_non_scriptable.py 2022-09-27T15:43:24.7893686Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:43:24.7902554Z 2022-09-27T15:43:24.7902697Z Running tests... 2022-09-27T15:43:24.7903255Z ---------------------------------------------------------------------- 2022-09-27T15:43:25.0287669Z test_async_function_wrong_return_type_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8803 2022-09-27T15:43:25.0314461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8804 2022-09-27T15:43:25.0342398Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8805 2022-09-27T15:43:25.0371360Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8806 2022-09-27T15:43:26.1500500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:26.1501128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:26.1713081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:26.1713743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:26.1885488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:26.1886030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:26.1963734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:26.1964537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:26.1974153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:26.1974979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:26.2112089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:26.2112697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:26.2355390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:26.2356051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:26.2365408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:26.2366039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:26.3158957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwbjib5hu 2022-09-27T15:43:26.3159780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwbjib5hu/_remote_module_non_scriptable.py 2022-09-27T15:43:26.3389107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpopzo33i7 2022-09-27T15:43:26.3389847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpopzo33i7/_remote_module_non_scriptable.py 2022-09-27T15:43:26.3628023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1q_joaeb 2022-09-27T15:43:26.3628717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1q_joaeb/_remote_module_non_scriptable.py 2022-09-27T15:43:26.3636738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpboz_yyps 2022-09-27T15:43:26.3638491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpboz_yyps/_remote_module_non_scriptable.py 2022-09-27T15:43:26.5754143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:26.5983796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:26.6230854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:26.6254961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:27.2426938Z ok (2.452s) 2022-09-27T15:43:27.2427127Z 2022-09-27T15:43:27.2427458Z ---------------------------------------------------------------------- 2022-09-27T15:43:27.2427694Z Ran 1 test in 2.452s 2022-09-27T15:43:27.2427807Z 2022-09-27T15:43:27.2427868Z OK 2022-09-27T15:43:27.2427958Z 2022-09-27T15:43:27.2428051Z Generating XML reports... 2022-09-27T15:43:27.2463020Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154324.xml 2022-09-27T15:43:28.6151081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:28.6151828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:28.6160763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:28.6161424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:28.7769030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5me6z5m1 2022-09-27T15:43:28.7769698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5me6z5m1/_remote_module_non_scriptable.py 2022-09-27T15:43:29.0382827Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:43:29.0393098Z 2022-09-27T15:43:29.0393519Z Running tests... 2022-09-27T15:43:29.0393962Z ---------------------------------------------------------------------- 2022-09-27T15:43:29.2849159Z test_async_record_function_cbs_jit_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9026 2022-09-27T15:43:29.2875813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9027 2022-09-27T15:43:29.2903909Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9028 2022-09-27T15:43:29.2933727Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9029 2022-09-27T15:43:30.3904990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:30.3905645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:30.4102249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:30.4102934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:30.4294857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:30.4295440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:30.4398186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:30.4398826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:30.4422366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:30.4423025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:30.4484728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:30.4485266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:30.4780837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:30.4781401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:30.4804125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:30.4804765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:30.5567425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphas1l_kj 2022-09-27T15:43:30.5568211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphas1l_kj/_remote_module_non_scriptable.py 2022-09-27T15:43:30.5751331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9lqv0lb 2022-09-27T15:43:30.5752065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9lqv0lb/_remote_module_non_scriptable.py 2022-09-27T15:43:30.6044600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdrteqxj 2022-09-27T15:43:30.6045290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdrteqxj/_remote_module_non_scriptable.py 2022-09-27T15:43:30.6074081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhzf0w1i 2022-09-27T15:43:30.6075810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhzf0w1i/_remote_module_non_scriptable.py 2022-09-27T15:43:30.8164887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:30.8529847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:30.8631958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:30.8686512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:31.4990599Z ok (2.459s) 2022-09-27T15:43:31.4990868Z 2022-09-27T15:43:31.4991708Z ---------------------------------------------------------------------- 2022-09-27T15:43:31.4991956Z Ran 1 test in 2.460s 2022-09-27T15:43:31.4992072Z 2022-09-27T15:43:31.4992132Z OK 2022-09-27T15:43:31.4992292Z 2022-09-27T15:43:31.4992390Z Generating XML reports... 2022-09-27T15:43:31.5026681Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154329.xml 2022-09-27T15:43:32.8912866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:32.8913251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:32.8922382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:32.8922939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:33.0526822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0xc3v18 2022-09-27T15:43:33.0528262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0xc3v18/_remote_module_non_scriptable.py 2022-09-27T15:43:33.3160014Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:43:33.3169396Z 2022-09-27T15:43:33.3169651Z Running tests... 2022-09-27T15:43:33.3170204Z ---------------------------------------------------------------------- 2022-09-27T15:43:33.5641450Z test_async_record_function_double_end_callbacks (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9249 2022-09-27T15:43:33.5668131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9250 2022-09-27T15:43:33.5695969Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9251 2022-09-27T15:43:33.5725916Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9252 2022-09-27T15:43:34.6887607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:34.6888307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:34.7250646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:34.7251234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:34.7315037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:34.7315692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:34.7673116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:34.7673744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:34.7798171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:34.7798821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:34.8042622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:34.8043323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:34.8158317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:34.8158854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:34.8398075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:34.8398746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:34.8517687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkrvxj49 2022-09-27T15:43:34.8519555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkrvxj49/_remote_module_non_scriptable.py 2022-09-27T15:43:34.8940555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptp3vavxi 2022-09-27T15:43:34.8941527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptp3vavxi/_remote_module_non_scriptable.py 2022-09-27T15:43:34.9428917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpno4ueg0p 2022-09-27T15:43:34.9429637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpno4ueg0p/_remote_module_non_scriptable.py 2022-09-27T15:43:34.9852371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphueywxw5 2022-09-27T15:43:34.9853074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphueywxw5/_remote_module_non_scriptable.py 2022-09-27T15:43:35.1082227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:35.1544944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:35.2029260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:35.2438942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:36.8799532Z ok (3.563s) 2022-09-27T15:43:36.8799757Z 2022-09-27T15:43:36.8800281Z ---------------------------------------------------------------------- 2022-09-27T15:43:36.8800747Z Ran 1 test in 3.563s 2022-09-27T15:43:36.8800937Z 2022-09-27T15:43:36.8801000Z OK 2022-09-27T15:43:36.8801080Z 2022-09-27T15:43:36.8801174Z Generating XML reports... 2022-09-27T15:43:36.8835988Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154333.xml 2022-09-27T15:43:38.2608054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:38.2608447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:38.2617743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:38.2618175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:38.4208186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1x9it0wq 2022-09-27T15:43:38.4209147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1x9it0wq/_remote_module_non_scriptable.py 2022-09-27T15:43:38.6837632Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:43:38.6847181Z 2022-09-27T15:43:38.6847507Z Running tests... 2022-09-27T15:43:38.6848115Z ---------------------------------------------------------------------- 2022-09-27T15:43:38.9288457Z test_async_record_function_double_end_callbacks_new_signatures (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9472 2022-09-27T15:43:38.9316680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9473 2022-09-27T15:43:38.9344469Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9474 2022-09-27T15:43:38.9373928Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9475 2022-09-27T15:43:40.0293279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:40.0293931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:40.0507223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:40.0507859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:40.0650451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:40.0651388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:40.0675362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:40.0676254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:40.0887444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:40.0888007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:40.0978095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:40.0978837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:40.1051838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:40.1052466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:40.1360666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:40.1361053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:40.1952038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzylho_3c 2022-09-27T15:43:40.1952849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzylho_3c/_remote_module_non_scriptable.py 2022-09-27T15:43:40.2327324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchmpb0yx 2022-09-27T15:43:40.2328393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppduyoy98 2022-09-27T15:43:40.2329155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchmpb0yx/_remote_module_non_scriptable.py 2022-09-27T15:43:40.2329921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppduyoy98/_remote_module_non_scriptable.py 2022-09-27T15:43:40.2623952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx12n7xs2 2022-09-27T15:43:40.2624847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx12n7xs2/_remote_module_non_scriptable.py 2022-09-27T15:43:40.4556190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:40.4891731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:40.4899839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:40.5197360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:42.1447223Z ok (3.460s) 2022-09-27T15:43:42.1447456Z 2022-09-27T15:43:42.1447959Z ---------------------------------------------------------------------- 2022-09-27T15:43:42.1448306Z Ran 1 test in 3.460s 2022-09-27T15:43:42.1448409Z 2022-09-27T15:43:42.1448467Z OK 2022-09-27T15:43:42.1448562Z 2022-09-27T15:43:42.1448670Z Generating XML reports... 2022-09-27T15:43:42.1482674Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154338.xml 2022-09-27T15:43:43.5069751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:43.5070126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:43.5078321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:43.5078764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:43.6664017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavrfp9xi 2022-09-27T15:43:43.6664916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavrfp9xi/_remote_module_non_scriptable.py 2022-09-27T15:43:43.9282045Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:43:43.9291306Z 2022-09-27T15:43:43.9291441Z Running tests... 2022-09-27T15:43:43.9292013Z ---------------------------------------------------------------------- 2022-09-27T15:43:44.1750311Z test_async_static_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9695 2022-09-27T15:43:44.1778421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9696 2022-09-27T15:43:44.1806427Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9697 2022-09-27T15:43:44.1835509Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9698 2022-09-27T15:43:45.3903521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:45.3904317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:45.4079100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:45.4079618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:45.4095861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:45.4096540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:45.4280157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:45.4280691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:45.4463445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:45.4464033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:45.4479975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:45.4480606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:45.4498763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:45.4499425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:45.4848977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:45.4849546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:45.5548622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpghhlzoki 2022-09-27T15:43:45.5549807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpghhlzoki/_remote_module_non_scriptable.py 2022-09-27T15:43:45.5732592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22mte4up 2022-09-27T15:43:45.5733408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22mte4up/_remote_module_non_scriptable.py 2022-09-27T15:43:45.5742805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67nuhnwe 2022-09-27T15:43:45.5745085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67nuhnwe/_remote_module_non_scriptable.py 2022-09-27T15:43:45.6107020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgist00rz 2022-09-27T15:43:45.6107765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgist00rz/_remote_module_non_scriptable.py 2022-09-27T15:43:45.8124978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:45.8311401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:45.8339862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:45.8670789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:46.5004820Z ok (2.571s) 2022-09-27T15:43:46.5005036Z 2022-09-27T15:43:46.5005435Z ---------------------------------------------------------------------- 2022-09-27T15:43:46.5006075Z Ran 1 test in 2.571s 2022-09-27T15:43:46.5006257Z 2022-09-27T15:43:46.5006310Z OK 2022-09-27T15:43:46.5006406Z 2022-09-27T15:43:46.5006507Z Generating XML reports... 2022-09-27T15:43:46.5042230Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154343.xml 2022-09-27T15:43:47.8743731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:47.8744136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:47.8751891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:47.8752448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:48.0337617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2bts9nn 2022-09-27T15:43:48.0338565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2bts9nn/_remote_module_non_scriptable.py 2022-09-27T15:43:48.2965484Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:43:48.2974832Z 2022-09-27T15:43:48.2975264Z Running tests... 2022-09-27T15:43:48.2975884Z ---------------------------------------------------------------------- 2022-09-27T15:43:48.5477482Z test_async_static_method_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9918 2022-09-27T15:43:48.5505389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9919 2022-09-27T15:43:48.5534348Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9920 2022-09-27T15:43:48.5572589Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9921 2022-09-27T15:43:49.6449044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:49.6449719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:49.6458176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:49.6486065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:49.6486925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:49.6487576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:49.6830002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:49.6830724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:49.6845960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:49.6846705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:49.6855998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:49.6856554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:49.6887642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:49.6888182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:49.7223468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:49.7224125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:49.8122372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4u5h37nv 2022-09-27T15:43:49.8123216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4u5h37nv/_remote_module_non_scriptable.py 2022-09-27T15:43:49.8127989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdodfylde 2022-09-27T15:43:49.8129944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdodfylde/_remote_module_non_scriptable.py 2022-09-27T15:43:49.8200200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeja2os28 2022-09-27T15:43:49.8201954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeja2os28/_remote_module_non_scriptable.py 2022-09-27T15:43:49.8482277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbue9kfo 2022-09-27T15:43:49.8483251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbue9kfo/_remote_module_non_scriptable.py 2022-09-27T15:43:50.0653823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:50.0660138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:50.0814339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:50.1025293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:50.7628971Z ok (2.465s) 2022-09-27T15:43:50.7629134Z 2022-09-27T15:43:50.7629451Z ---------------------------------------------------------------------- 2022-09-27T15:43:50.7629702Z Ran 1 test in 2.465s 2022-09-27T15:43:50.7629816Z 2022-09-27T15:43:50.7629875Z OK 2022-09-27T15:43:50.7629965Z 2022-09-27T15:43:50.7630058Z Generating XML reports... 2022-09-27T15:43:50.7665749Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154348.xml 2022-09-27T15:43:52.1424765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:52.1425479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:52.1434144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:52.1435064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:52.3028714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpustumnrp 2022-09-27T15:43:52.3029563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpustumnrp/_remote_module_non_scriptable.py 2022-09-27T15:43:52.5663994Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:43:52.5673475Z 2022-09-27T15:43:52.5673690Z Running tests... 2022-09-27T15:43:52.5674063Z ---------------------------------------------------------------------- 2022-09-27T15:43:52.8113125Z test_build_rpc_profiling_key (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10141 2022-09-27T15:43:52.8140890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10142 2022-09-27T15:43:52.8168517Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10143 2022-09-27T15:43:52.8199061Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10144 2022-09-27T15:43:53.8924863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:53.8925544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:53.8935560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:53.8936153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:53.9304612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:53.9305301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:53.9313445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:53.9314046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:53.9396839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:53.9397469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:53.9596801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:53.9597465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:53.9763240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:53.9763915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:53.9955917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:53.9956583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:54.0582292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4uzuom4y 2022-09-27T15:43:54.0583292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4uzuom4y/_remote_module_non_scriptable.py 2022-09-27T15:43:54.0587099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgev3_db3 2022-09-27T15:43:54.0589485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgev3_db3/_remote_module_non_scriptable.py 2022-09-27T15:43:54.1030731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2wcagqkm 2022-09-27T15:43:54.1031847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2wcagqkm/_remote_module_non_scriptable.py 2022-09-27T15:43:54.1208153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5fbb50hl 2022-09-27T15:43:54.1209125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5fbb50hl/_remote_module_non_scriptable.py 2022-09-27T15:43:54.3174062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:54.3212278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:54.3636358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:54.3794287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:54.7247969Z ok (2.157s) 2022-09-27T15:43:54.7248235Z 2022-09-27T15:43:54.7248745Z ---------------------------------------------------------------------- 2022-09-27T15:43:54.7249183Z Ran 1 test in 2.157s 2022-09-27T15:43:54.7249413Z 2022-09-27T15:43:54.7249499Z OK 2022-09-27T15:43:54.7249659Z 2022-09-27T15:43:54.7249813Z Generating XML reports... 2022-09-27T15:43:54.7286461Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154352.xml 2022-09-27T15:43:56.0948595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:56.0948978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:56.0957011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:56.0957651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:56.2508294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xh8q2wf 2022-09-27T15:43:56.2509396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xh8q2wf/_remote_module_non_scriptable.py 2022-09-27T15:43:56.5105418Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:43:56.5114819Z 2022-09-27T15:43:56.5114978Z Running tests... 2022-09-27T15:43:56.5115586Z ---------------------------------------------------------------------- 2022-09-27T15:43:56.7687079Z test_builtin_remote_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10196 2022-09-27T15:43:56.7715154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10197 2022-09-27T15:43:56.7743242Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10198 2022-09-27T15:43:56.7772717Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10199 2022-09-27T15:43:57.9141800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:57.9142539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:57.9300095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:57.9300660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:57.9380443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:57.9380987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:57.9524025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:57.9524650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:57.9680797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:57.9681475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:57.9749842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:43:57.9750374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:43:57.9752852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:57.9753430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:58.0107823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:43:58.0108452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:43:58.0795613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptuwtu3k3 2022-09-27T15:43:58.0796448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptuwtu3k3/_remote_module_non_scriptable.py 2022-09-27T15:43:58.0938293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8tske3c 2022-09-27T15:43:58.0939457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8tske3c/_remote_module_non_scriptable.py 2022-09-27T15:43:58.1003417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4etz_hr 2022-09-27T15:43:58.1004482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4etz_hr/_remote_module_non_scriptable.py 2022-09-27T15:43:58.1364455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphja4sszw 2022-09-27T15:43:58.1365211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphja4sszw/_remote_module_non_scriptable.py 2022-09-27T15:43:58.3374517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:43:58.3487929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:43:58.3547974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:43:58.3928350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:43:59.0831142Z ok (2.571s) 2022-09-27T15:43:59.0831470Z 2022-09-27T15:43:59.0831993Z ---------------------------------------------------------------------- 2022-09-27T15:43:59.0832302Z Ran 1 test in 2.572s 2022-09-27T15:43:59.0832419Z 2022-09-27T15:43:59.0832466Z OK 2022-09-27T15:43:59.0832560Z 2022-09-27T15:43:59.0832652Z Generating XML reports... 2022-09-27T15:43:59.0867762Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154356.xml 2022-09-27T15:44:00.4682831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:00.4683258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:00.4692015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:00.4692419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:00.6259852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4oy6u4ts 2022-09-27T15:44:00.6260995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4oy6u4ts/_remote_module_non_scriptable.py 2022-09-27T15:44:00.8887440Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:00.8896766Z 2022-09-27T15:44:00.8896881Z Running tests... 2022-09-27T15:44:00.8897267Z ---------------------------------------------------------------------- 2022-09-27T15:44:01.1344674Z test_builtin_remote_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10419 2022-09-27T15:44:01.1372956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10420 2022-09-27T15:44:01.1400596Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10421 2022-09-27T15:44:01.1431854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10422 2022-09-27T15:44:02.2467902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:02.2468579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:02.2610236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:02.2610916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:02.2805356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:02.2805855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:02.2867746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:02.2868296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:02.3005606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:02.3006176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:02.3108253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:02.3108873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:02.3192146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:02.3192773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:02.3482666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:02.3483667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:02.4158467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2wwe7le 2022-09-27T15:44:02.4159508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2wwe7le/_remote_module_non_scriptable.py 2022-09-27T15:44:02.4276526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppq5d3dyn 2022-09-27T15:44:02.4277566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppq5d3dyn/_remote_module_non_scriptable.py 2022-09-27T15:44:02.4462229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppff09pyj 2022-09-27T15:44:02.4462897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppff09pyj/_remote_module_non_scriptable.py 2022-09-27T15:44:02.4739998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwt343gig 2022-09-27T15:44:02.4741410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwt343gig/_remote_module_non_scriptable.py 2022-09-27T15:44:02.6783763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:44:02.6835759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:02.7022438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:02.7271798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:03.3487254Z ok (2.459s) 2022-09-27T15:44:03.3487489Z 2022-09-27T15:44:03.3488109Z ---------------------------------------------------------------------- 2022-09-27T15:44:03.3488540Z Ran 1 test in 2.459s 2022-09-27T15:44:03.3488666Z 2022-09-27T15:44:03.3488715Z OK 2022-09-27T15:44:03.3488805Z 2022-09-27T15:44:03.3488899Z Generating XML reports... 2022-09-27T15:44:03.3526872Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154400.xml 2022-09-27T15:44:04.7047012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:04.7047417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:04.7054824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:04.8636058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:04.8636528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd__je8vq 2022-09-27T15:44:04.8637269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd__je8vq/_remote_module_non_scriptable.py 2022-09-27T15:44:05.1310086Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:05.1320561Z 2022-09-27T15:44:05.1320949Z Running tests... 2022-09-27T15:44:05.1321663Z ---------------------------------------------------------------------- 2022-09-27T15:44:05.1331530Z test_call_method_on_rref (__main__.TensorPipeRpcTest) 2022-09-27T15:44:05.3743899Z Tests that it is possible to call an instance method on a remote objet ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10642 2022-09-27T15:44:05.3771311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10643 2022-09-27T15:44:05.3798953Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10644 2022-09-27T15:44:05.3828918Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10645 2022-09-27T15:44:06.5378575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:06.5379207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:06.5413254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:06.5413872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:06.5514417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:06.5515600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:06.5676570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:06.5677229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:06.5759224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:06.5759749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:06.5800385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:06.5801036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:06.5899395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:06.5900043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:06.6046624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:06.6047287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:06.7029486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppej8dflp 2022-09-27T15:44:06.7030217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppej8dflp/_remote_module_non_scriptable.py 2022-09-27T15:44:06.7069196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2krtutw0 2022-09-27T15:44:06.7070565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2krtutw0/_remote_module_non_scriptable.py 2022-09-27T15:44:06.7165013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4e6hwd5n 2022-09-27T15:44:06.7166357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4e6hwd5n/_remote_module_non_scriptable.py 2022-09-27T15:44:06.7313308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8f1pzfj 2022-09-27T15:44:06.7314094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8f1pzfj/_remote_module_non_scriptable.py 2022-09-27T15:44:06.9575084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:06.9643546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:06.9726189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:44:06.9868431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:07.6888136Z ok (2.556s) 2022-09-27T15:44:07.6888367Z 2022-09-27T15:44:07.6888787Z ---------------------------------------------------------------------- 2022-09-27T15:44:07.6889204Z Ran 1 test in 2.557s 2022-09-27T15:44:07.6889370Z 2022-09-27T15:44:07.6889452Z OK 2022-09-27T15:44:07.6889616Z 2022-09-27T15:44:07.6889766Z Generating XML reports... 2022-09-27T15:44:07.6926383Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154405.xml 2022-09-27T15:44:09.0536626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:09.0537075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:09.0545903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:09.0546263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:09.2127714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jfuxkk1 2022-09-27T15:44:09.2128466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jfuxkk1/_remote_module_non_scriptable.py 2022-09-27T15:44:09.4748865Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:09.4758381Z 2022-09-27T15:44:09.4758501Z Running tests... 2022-09-27T15:44:09.4759086Z ---------------------------------------------------------------------- 2022-09-27T15:44:09.7208589Z test_callback_chain (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10865 2022-09-27T15:44:09.7235639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10866 2022-09-27T15:44:09.7262980Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10867 2022-09-27T15:44:09.7292020Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10868 2022-09-27T15:44:10.8076044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:10.8076679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:10.8077442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:10.8078031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:10.8271278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:10.8272022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:10.8336983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:10.8337517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:10.8462753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:10.8463326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:10.8463962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:10.8464449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:10.8656303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:10.8656966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:10.8718865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:10.8719484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:10.9728753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlqppkfd 2022-09-27T15:44:10.9729566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlqppkfd/_remote_module_non_scriptable.py 2022-09-27T15:44:10.9730671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_ndgcfx 2022-09-27T15:44:10.9733766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_ndgcfx/_remote_module_non_scriptable.py 2022-09-27T15:44:10.9914958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5zgi0d_z 2022-09-27T15:44:10.9916109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5zgi0d_z/_remote_module_non_scriptable.py 2022-09-27T15:44:10.9975770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuh_esuk4 2022-09-27T15:44:10.9977741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuh_esuk4/_remote_module_non_scriptable.py 2022-09-27T15:44:11.2309145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:11.2329668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:11.2490530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:44:11.2739938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:11.9349489Z ok (2.459s) 2022-09-27T15:44:11.9349733Z 2022-09-27T15:44:11.9350260Z ---------------------------------------------------------------------- 2022-09-27T15:44:11.9350526Z Ran 1 test in 2.459s 2022-09-27T15:44:11.9350643Z 2022-09-27T15:44:11.9350704Z OK 2022-09-27T15:44:11.9350795Z 2022-09-27T15:44:11.9350878Z Generating XML reports... 2022-09-27T15:44:11.9384976Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154409.xml 2022-09-27T15:44:13.3144792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:13.3145165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:13.3153417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:13.3153788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:13.4703602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbwgha1l 2022-09-27T15:44:13.4704744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbwgha1l/_remote_module_non_scriptable.py 2022-09-27T15:44:13.7357731Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:13.7367605Z 2022-09-27T15:44:13.7367745Z Running tests... 2022-09-27T15:44:13.7368193Z ---------------------------------------------------------------------- 2022-09-27T15:44:13.9869017Z test_callback_in_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11088 2022-09-27T15:44:13.9896691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11089 2022-09-27T15:44:13.9925273Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11090 2022-09-27T15:44:13.9956127Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11091 2022-09-27T15:44:15.1147896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:15.1148553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:15.1225494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:15.1226064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:15.1529205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:15.1529829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:15.1602080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:15.1602757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:15.1711968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:15.1712567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:15.1815358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:15.1815997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:15.2074949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:15.2087218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:15.2174764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:15.2175783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:15.2797946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcet7rhcp 2022-09-27T15:44:15.2798967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcet7rhcp/_remote_module_non_scriptable.py 2022-09-27T15:44:15.2866239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfo5femkj 2022-09-27T15:44:15.2867224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfo5femkj/_remote_module_non_scriptable.py 2022-09-27T15:44:15.3337809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8gpuw44 2022-09-27T15:44:15.3338586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8gpuw44/_remote_module_non_scriptable.py 2022-09-27T15:44:15.3612935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbv1pokc5 2022-09-27T15:44:15.3613770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbv1pokc5/_remote_module_non_scriptable.py 2022-09-27T15:44:15.5363081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:15.5454268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:15.5936828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:44:15.6202761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:16.2011455Z ok (2.464s) 2022-09-27T15:44:16.2011805Z 2022-09-27T15:44:16.2012316Z ---------------------------------------------------------------------- 2022-09-27T15:44:16.2012624Z Ran 1 test in 2.464s 2022-09-27T15:44:16.2012740Z 2022-09-27T15:44:16.2012803Z OK 2022-09-27T15:44:16.2012893Z 2022-09-27T15:44:16.2012995Z Generating XML reports... 2022-09-27T15:44:16.2049861Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154413.xml 2022-09-27T15:44:17.5614685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:17.5615077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:17.5621996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:17.5622351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:17.7212520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6azl_9x 2022-09-27T15:44:17.7213797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6azl_9x/_remote_module_non_scriptable.py 2022-09-27T15:44:17.9835016Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:17.9848000Z 2022-09-27T15:44:17.9848134Z Running tests... 2022-09-27T15:44:17.9848669Z ---------------------------------------------------------------------- 2022-09-27T15:44:18.2319792Z test_callback_multi (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11311 2022-09-27T15:44:18.2348284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11312 2022-09-27T15:44:18.2376127Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11313 2022-09-27T15:44:18.2407211Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11314 2022-09-27T15:44:19.3884807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:19.3885490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:19.4170744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:19.4171488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:19.4249476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:19.4250118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:19.4427960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:19.4428565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:19.4437156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:19.4437818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:19.4558655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:19.4559199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:19.4810524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:19.4811147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:19.4825042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:19.4825663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:19.5682727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5sd03zf 2022-09-27T15:44:19.5683557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5sd03zf/_remote_module_non_scriptable.py 2022-09-27T15:44:19.5850704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5kc1h8t 2022-09-27T15:44:19.5851497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5kc1h8t/_remote_module_non_scriptable.py 2022-09-27T15:44:19.6081751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxk2gzm5w 2022-09-27T15:44:19.6082458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxk2gzm5w/_remote_module_non_scriptable.py 2022-09-27T15:44:19.6100135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85a3emf_ 2022-09-27T15:44:19.6102861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85a3emf_/_remote_module_non_scriptable.py 2022-09-27T15:44:19.8245442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:44:19.8404727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:19.8633783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:19.8694368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:20.5464658Z ok (2.561s) 2022-09-27T15:44:20.5464846Z 2022-09-27T15:44:20.5465172Z ---------------------------------------------------------------------- 2022-09-27T15:44:20.5465471Z Ran 1 test in 2.562s 2022-09-27T15:44:20.5465590Z 2022-09-27T15:44:20.5465651Z OK 2022-09-27T15:44:20.5465745Z 2022-09-27T15:44:20.5465839Z Generating XML reports... 2022-09-27T15:44:20.5501019Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154417.xml 2022-09-27T15:44:21.9387749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:21.9388136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:21.9396306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:21.9397022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:22.1005270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpik4_hne0 2022-09-27T15:44:22.1006418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpik4_hne0/_remote_module_non_scriptable.py 2022-09-27T15:44:22.3604613Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:22.3613389Z 2022-09-27T15:44:22.3613523Z Running tests... 2022-09-27T15:44:22.3614087Z ---------------------------------------------------------------------- 2022-09-27T15:44:22.6026407Z test_callback_none (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11534 2022-09-27T15:44:22.6053358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11535 2022-09-27T15:44:22.6080659Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11536 2022-09-27T15:44:22.6110820Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11537 2022-09-27T15:44:23.7396882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:23.7397466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:23.7697105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:23.7697740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:23.7714050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:23.7714825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:23.7771488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:23.7772197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:23.8068407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:23.8069016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:23.8088792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:23.8089440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:23.8232862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:23.8233531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:23.8583945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:23.8584602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:23.9048941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_lt7tra 2022-09-27T15:44:23.9049738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_lt7tra/_remote_module_non_scriptable.py 2022-09-27T15:44:23.9338826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4krrz_re 2022-09-27T15:44:23.9339617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4krrz_re/_remote_module_non_scriptable.py 2022-09-27T15:44:23.9362089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3or3jowz 2022-09-27T15:44:23.9363436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3or3jowz/_remote_module_non_scriptable.py 2022-09-27T15:44:23.9842173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05vzg4bv 2022-09-27T15:44:23.9843625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05vzg4bv/_remote_module_non_scriptable.py 2022-09-27T15:44:24.1639906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:44:24.1889689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:24.1931956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:24.2417732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:24.4629464Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:44:24.4630440Z ValueError('Expected error') 2022-09-27T15:44:24.4631048Z Traceback (most recent call last): 2022-09-27T15:44:24.4632191Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:44:24.4633108Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:44:24.4634225Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:44:24.4635121Z raise ValueError(expected_err) 2022-09-27T15:44:24.4635571Z ValueError: Expected error 2022-09-27T15:44:24.4635850Z 2022-09-27T15:44:24.4786704Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:44:24.4787435Z ValueError('Expected error') 2022-09-27T15:44:24.4787875Z Traceback (most recent call last): 2022-09-27T15:44:24.4790771Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:44:24.4791706Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:44:24.4793027Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:44:24.4794065Z raise ValueError(expected_err) 2022-09-27T15:44:24.4794690Z ValueError: Expected error 2022-09-27T15:44:24.4794955Z 2022-09-27T15:44:24.4836266Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:44:24.4836769Z ValueError('Expected error') 2022-09-27T15:44:24.4837510Z Traceback (most recent call last): 2022-09-27T15:44:24.4838205Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:44:24.4838798Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:44:24.4839573Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:44:24.4840095Z raise ValueError(expected_err) 2022-09-27T15:44:24.4840439Z ValueError: Expected error 2022-09-27T15:44:24.4840662Z 2022-09-27T15:44:24.4888963Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:44:24.4889628Z ValueError('Expected error') 2022-09-27T15:44:24.4890078Z Traceback (most recent call last): 2022-09-27T15:44:24.4890944Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:44:24.4891680Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:44:24.4892683Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:44:24.4893336Z raise ValueError(expected_err) 2022-09-27T15:44:24.4893786Z ValueError: Expected error 2022-09-27T15:44:24.4894051Z 2022-09-27T15:44:24.9168403Z ok (2.555s) 2022-09-27T15:44:24.9168564Z 2022-09-27T15:44:24.9168928Z ---------------------------------------------------------------------- 2022-09-27T15:44:24.9169181Z Ran 1 test in 2.555s 2022-09-27T15:44:24.9169299Z 2022-09-27T15:44:24.9169360Z OK 2022-09-27T15:44:24.9169454Z 2022-09-27T15:44:24.9169548Z Generating XML reports... 2022-09-27T15:44:24.9204902Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154422.xml 2022-09-27T15:44:26.3042374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:26.3043015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:26.3050566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:26.3051039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:26.4651008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1m1fmk52 2022-09-27T15:44:26.4652168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1m1fmk52/_remote_module_non_scriptable.py 2022-09-27T15:44:26.7286139Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:26.7296397Z 2022-09-27T15:44:26.7296522Z Running tests... 2022-09-27T15:44:26.7297094Z ---------------------------------------------------------------------- 2022-09-27T15:44:26.9728335Z test_callback_simple (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11757 2022-09-27T15:44:26.9755852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11758 2022-09-27T15:44:26.9783652Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11759 2022-09-27T15:44:26.9813907Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11760 2022-09-27T15:44:28.1273234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:28.1273877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:28.1638748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:28.1639331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:28.1663718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:28.1664405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:28.1838874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:28.1839406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:28.2048549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:28.2049101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:28.2132004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:28.2132632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:28.2221538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:28.2222168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:28.2510593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:28.2511329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:28.2911842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpejxhpf 2022-09-27T15:44:28.2912792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpejxhpf/_remote_module_non_scriptable.py 2022-09-27T15:44:28.3333536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3xeyyas6 2022-09-27T15:44:28.3334296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3xeyyas6/_remote_module_non_scriptable.py 2022-09-27T15:44:28.3498108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19r7lurz 2022-09-27T15:44:28.3498913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19r7lurz/_remote_module_non_scriptable.py 2022-09-27T15:44:28.3779865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzzma7th 2022-09-27T15:44:28.3780624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzzma7th/_remote_module_non_scriptable.py 2022-09-27T15:44:28.5559148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:28.6014379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:44:28.6147908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:28.6418311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:29.2872288Z ok (2.557s) 2022-09-27T15:44:29.2872474Z 2022-09-27T15:44:29.2872774Z ---------------------------------------------------------------------- 2022-09-27T15:44:29.2873083Z Ran 1 test in 2.557s 2022-09-27T15:44:29.2873198Z 2022-09-27T15:44:29.2873262Z OK 2022-09-27T15:44:29.2873356Z 2022-09-27T15:44:29.2873465Z Generating XML reports... 2022-09-27T15:44:29.2907474Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154426.xml 2022-09-27T15:44:30.6684106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:30.6684492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:30.6693446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:30.6693983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:30.8309094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifa67iwd 2022-09-27T15:44:30.8309760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifa67iwd/_remote_module_non_scriptable.py 2022-09-27T15:44:31.0925172Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:31.0934744Z 2022-09-27T15:44:31.0934871Z Running tests... 2022-09-27T15:44:31.0935315Z ---------------------------------------------------------------------- 2022-09-27T15:44:31.3371101Z test_callback_with_error (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11980 2022-09-27T15:44:31.3399617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11981 2022-09-27T15:44:31.3427089Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11982 2022-09-27T15:44:31.3457243Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11983 2022-09-27T15:44:32.5115287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:32.5115958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:32.5378133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:32.5378823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:32.5379620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:32.5380249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:32.5477947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:32.5478488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:32.5738683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:32.5739397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:32.5757612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:32.5758242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:32.5760677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:32.5761431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:32.6100472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:32.6101146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:32.6746024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9jrtwhg 2022-09-27T15:44:32.6747466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9jrtwhg/_remote_module_non_scriptable.py 2022-09-27T15:44:32.7023851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2cih_g0 2022-09-27T15:44:32.7024645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2cih_g0/_remote_module_non_scriptable.py 2022-09-27T15:44:32.7033169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxi0_1ty6 2022-09-27T15:44:32.7035057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxi0_1ty6/_remote_module_non_scriptable.py 2022-09-27T15:44:32.7363677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6t7uwq4 2022-09-27T15:44:32.7364422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6t7uwq4/_remote_module_non_scriptable.py 2022-09-27T15:44:32.9330177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:44:32.9589218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:32.9651409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:32.9938686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:33.1943127Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:44:33.1943891Z ValueError('Expected error') 2022-09-27T15:44:33.1944355Z Traceback (most recent call last): 2022-09-27T15:44:33.1945241Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:44:33.1945973Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:44:33.1946933Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:44:33.1947561Z raise ValueError(expected_err) 2022-09-27T15:44:33.1948005Z ValueError: Expected error 2022-09-27T15:44:33.1948270Z 2022-09-27T15:44:33.2144019Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:44:33.2144807Z ValueError('Expected error') 2022-09-27T15:44:33.2145437Z Traceback (most recent call last): 2022-09-27T15:44:33.2146509Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:44:33.2147547Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:44:33.2148625Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:44:33.2149276Z raise ValueError(expected_err) 2022-09-27T15:44:33.2149725Z ValueError: Expected error 2022-09-27T15:44:33.2149993Z 2022-09-27T15:44:33.2225838Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:44:33.2226390Z ValueError('Expected error') 2022-09-27T15:44:33.2226780Z Traceback (most recent call last): 2022-09-27T15:44:33.2227502Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:44:33.2228054Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:44:33.2229128Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:44:33.2229659Z raise ValueError(expected_err) 2022-09-27T15:44:33.2230009Z ValueError: Expected error 2022-09-27T15:44:33.2230328Z 2022-09-27T15:44:33.2254381Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:44:33.2255309Z ValueError('Expected error') 2022-09-27T15:44:33.2255718Z Traceback (most recent call last): 2022-09-27T15:44:33.2256375Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:44:33.2256722Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:44:33.2257205Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:44:33.2257514Z raise ValueError(expected_err) 2022-09-27T15:44:33.2257726Z ValueError: Expected error 2022-09-27T15:44:33.2257839Z 2022-09-27T15:44:33.6519528Z ok (2.558s) 2022-09-27T15:44:33.6519788Z 2022-09-27T15:44:33.6520309Z ---------------------------------------------------------------------- 2022-09-27T15:44:33.6520695Z Ran 1 test in 2.558s 2022-09-27T15:44:33.6520849Z 2022-09-27T15:44:33.6520910Z OK 2022-09-27T15:44:33.6521001Z 2022-09-27T15:44:33.6521096Z Generating XML reports... 2022-09-27T15:44:33.6557561Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154431.xml 2022-09-27T15:44:35.0404877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:35.0405275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:35.0412453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:35.0412818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:35.1959816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcem18put 2022-09-27T15:44:35.1960861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcem18put/_remote_module_non_scriptable.py 2022-09-27T15:44:35.4569489Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:35.4578646Z 2022-09-27T15:44:35.4578744Z Running tests... 2022-09-27T15:44:35.4579584Z ---------------------------------------------------------------------- 2022-09-27T15:44:35.6996041Z test_callback_with_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12203 2022-09-27T15:44:35.7024289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12204 2022-09-27T15:44:35.7052156Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12205 2022-09-27T15:44:35.7081423Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12206 2022-09-27T15:44:36.8483575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:36.8484255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:36.8654608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:36.8655272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:36.8841394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:36.8842028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:36.8859302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:36.8859936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:36.9028052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:36.9028942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:36.9051629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:36.9052484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:36.9236410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:36.9236942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:36.9426186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:36.9426840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:37.0113361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqit07hp 2022-09-27T15:44:37.0114162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqit07hp/_remote_module_non_scriptable.py 2022-09-27T15:44:37.0295135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0x9q5iy 2022-09-27T15:44:37.0296174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0x9q5iy/_remote_module_non_scriptable.py 2022-09-27T15:44:37.0496134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp803td14q 2022-09-27T15:44:37.0497939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp803td14q/_remote_module_non_scriptable.py 2022-09-27T15:44:37.0682253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9514zq5m 2022-09-27T15:44:37.0682981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9514zq5m/_remote_module_non_scriptable.py 2022-09-27T15:44:37.2705825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:37.2867335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:44:37.3222508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:37.3290597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:38.0139807Z ok (2.556s) 2022-09-27T15:44:38.0140069Z 2022-09-27T15:44:38.0140530Z ---------------------------------------------------------------------- 2022-09-27T15:44:38.0140803Z Ran 1 test in 2.556s 2022-09-27T15:44:38.0140920Z 2022-09-27T15:44:38.0140968Z OK 2022-09-27T15:44:38.0141062Z 2022-09-27T15:44:38.0141160Z Generating XML reports... 2022-09-27T15:44:38.0176121Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154435.xml 2022-09-27T15:44:39.3825084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:39.3825452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:39.3833727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:39.3834237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:39.5424922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5735yndj 2022-09-27T15:44:39.5425939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5735yndj/_remote_module_non_scriptable.py 2022-09-27T15:44:39.8056966Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:39.8066601Z 2022-09-27T15:44:39.8066725Z Running tests... 2022-09-27T15:44:39.8067297Z ---------------------------------------------------------------------- 2022-09-27T15:44:40.0486516Z test_callback_wrong_arg_num (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12426 2022-09-27T15:44:40.0512964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12427 2022-09-27T15:44:40.0540475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12428 2022-09-27T15:44:40.0569811Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12429 2022-09-27T15:44:41.1944916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:41.1945595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:41.1982787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:41.1983466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:41.2327073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:41.2327697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:41.2360632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:41.2361576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:41.2537161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:41.2537772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:41.2656781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:41.2657393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:41.2903850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:41.2904536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:41.3027684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:41.3028320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:41.3613118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwp0bo_o0 2022-09-27T15:44:41.3613932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwp0bo_o0/_remote_module_non_scriptable.py 2022-09-27T15:44:41.3629593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56s8dyw_ 2022-09-27T15:44:41.3631305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56s8dyw_/_remote_module_non_scriptable.py 2022-09-27T15:44:41.4160819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1mo7xpvg 2022-09-27T15:44:41.4161526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1mo7xpvg/_remote_module_non_scriptable.py 2022-09-27T15:44:41.4286278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4fql9g7 2022-09-27T15:44:41.4287112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4fql9g7/_remote_module_non_scriptable.py 2022-09-27T15:44:41.6177653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:44:41.6187996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:41.6723809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:41.6846023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:42.3628875Z ok (2.556s) 2022-09-27T15:44:42.3629120Z 2022-09-27T15:44:42.3629626Z ---------------------------------------------------------------------- 2022-09-27T15:44:42.3629896Z Ran 1 test in 2.556s 2022-09-27T15:44:42.3630014Z 2022-09-27T15:44:42.3630076Z OK 2022-09-27T15:44:42.3630168Z 2022-09-27T15:44:42.3630250Z Generating XML reports... 2022-09-27T15:44:42.3665549Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154439.xml 2022-09-27T15:44:43.7304173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:43.7304798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:43.7313230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:43.7313620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:43.8879866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ney0pnh 2022-09-27T15:44:43.8880734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ney0pnh/_remote_module_non_scriptable.py 2022-09-27T15:44:44.1476066Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:44.1485994Z 2022-09-27T15:44:44.1486211Z Running tests... 2022-09-27T15:44:44.1486807Z ---------------------------------------------------------------------- 2022-09-27T15:44:44.3938998Z test_callback_wrong_arg_type (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12649 2022-09-27T15:44:44.3966034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12650 2022-09-27T15:44:44.3993915Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12651 2022-09-27T15:44:44.4030150Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12652 2022-09-27T15:44:45.5425040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:45.5425658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:45.5431833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:45.5432417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:45.5435172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:45.5435738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:45.5525237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:45.5525759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:45.5808424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:45.5809161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:45.5816604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:45.5817171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:45.5822710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:45.5823299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:45.5905723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:45.5906284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:45.7075031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoaft7333 2022-09-27T15:44:45.7075853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoaft7333/_remote_module_non_scriptable.py 2022-09-27T15:44:45.7081327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46xdel63 2022-09-27T15:44:45.7083028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46xdel63/_remote_module_non_scriptable.py 2022-09-27T15:44:45.7084355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwg5vf78v 2022-09-27T15:44:45.7086780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwg5vf78v/_remote_module_non_scriptable.py 2022-09-27T15:44:45.7160510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhbm2fml 2022-09-27T15:44:45.7162799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhbm2fml/_remote_module_non_scriptable.py 2022-09-27T15:44:45.9609316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:45.9635203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:45.9639082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:45.9699986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:44:46.6086514Z ok (2.460s) 2022-09-27T15:44:46.6086881Z 2022-09-27T15:44:46.6087395Z ---------------------------------------------------------------------- 2022-09-27T15:44:46.6087860Z Ran 1 test in 2.460s 2022-09-27T15:44:46.6088052Z 2022-09-27T15:44:46.6088164Z OK 2022-09-27T15:44:46.6088327Z 2022-09-27T15:44:46.6088492Z Generating XML reports... 2022-09-27T15:44:46.6122841Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154444.xml 2022-09-27T15:44:47.9605646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:47.9606012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:47.9614052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:47.9614415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:48.1206487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjdpvnfqp 2022-09-27T15:44:48.1207426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjdpvnfqp/_remote_module_non_scriptable.py 2022-09-27T15:44:48.3815958Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:48.3825733Z 2022-09-27T15:44:48.3825825Z Running tests... 2022-09-27T15:44:48.3826363Z ---------------------------------------------------------------------- 2022-09-27T15:44:48.6235307Z test_cannot_infer_backend_from_options (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12872 2022-09-27T15:44:48.6263438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12873 2022-09-27T15:44:48.6291298Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12874 2022-09-27T15:44:48.6320662Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12875 2022-09-27T15:44:49.8233568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:49.8234244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:49.8370248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:49.8370917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:49.8445560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:49.8446114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:49.8458282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:49.8458897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:49.8623244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:49.8623877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:49.8758530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:49.8759203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:49.8840453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:49.8841083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:49.8850344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:49.8850912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:49.9897310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5sgfum65 2022-09-27T15:44:49.9898121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5sgfum65/_remote_module_non_scriptable.py 2022-09-27T15:44:50.0020570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbsdm7yuq 2022-09-27T15:44:50.0021368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbsdm7yuq/_remote_module_non_scriptable.py 2022-09-27T15:44:50.0101251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphej1g317 2022-09-27T15:44:50.0103006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphej1g317/_remote_module_non_scriptable.py 2022-09-27T15:44:50.0108484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0s74gpo 2022-09-27T15:44:50.0110799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0s74gpo/_remote_module_non_scriptable.py 2022-09-27T15:44:50.2472859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:44:50.2580778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:50.2643641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:50.2678087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:50.6373925Z ok (2.254s) 2022-09-27T15:44:50.6374162Z 2022-09-27T15:44:50.6374669Z ---------------------------------------------------------------------- 2022-09-27T15:44:50.6375020Z Ran 1 test in 2.255s 2022-09-27T15:44:50.6375137Z 2022-09-27T15:44:50.6375184Z OK 2022-09-27T15:44:50.6375278Z 2022-09-27T15:44:50.6375369Z Generating XML reports... 2022-09-27T15:44:50.6410295Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154448.xml 2022-09-27T15:44:52.0048213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:52.0048600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:52.0056935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:52.0057305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:52.1640819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_t_sorbg 2022-09-27T15:44:52.1641867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_t_sorbg/_remote_module_non_scriptable.py 2022-09-27T15:44:52.4252372Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:52.4262145Z 2022-09-27T15:44:52.4262249Z Running tests... 2022-09-27T15:44:52.4263119Z ---------------------------------------------------------------------- 2022-09-27T15:44:52.6727618Z test_deadlock (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12927 2022-09-27T15:44:52.6755698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12928 2022-09-27T15:44:52.6782708Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12929 2022-09-27T15:44:52.6812636Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12930 2022-09-27T15:44:53.8051657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:53.8052343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:53.8136633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:53.8137316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:53.8320470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:53.8321029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:53.8420387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:53.8421016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:53.8425468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:53.8426108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:53.8520085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:53.8520703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:53.8697578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:53.8698237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:53.8793233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:53.8793903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:53.9692838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpps5ezcjb 2022-09-27T15:44:53.9693603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpps5ezcjb/_remote_module_non_scriptable.py 2022-09-27T15:44:53.9781587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyqip17b 2022-09-27T15:44:53.9782349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyqip17b/_remote_module_non_scriptable.py 2022-09-27T15:44:53.9958057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xwfb0g8 2022-09-27T15:44:53.9959247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xwfb0g8/_remote_module_non_scriptable.py 2022-09-27T15:44:54.0060864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplrknbum8 2022-09-27T15:44:54.0062452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplrknbum8/_remote_module_non_scriptable.py 2022-09-27T15:44:54.2242851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:54.2321274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:54.2497588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:44:54.2639113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:55.5329096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:44:55.5366218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:44:55.5468745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:44:55.5470137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:44:55.5471449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:44:55.5472606Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:44:55.5473709Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:44:55.5532702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:44:55.9886181Z ok (3.562s) 2022-09-27T15:44:55.9886394Z 2022-09-27T15:44:55.9886747Z ---------------------------------------------------------------------- 2022-09-27T15:44:55.9886999Z Ran 1 test in 3.562s 2022-09-27T15:44:55.9887118Z 2022-09-27T15:44:55.9887179Z OK 2022-09-27T15:44:55.9887265Z 2022-09-27T15:44:55.9887360Z Generating XML reports... 2022-09-27T15:44:55.9924739Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154452.xml 2022-09-27T15:44:57.3768788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:57.3769472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:57.3777678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:57.3778295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:57.5346078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3zv_rtn_ 2022-09-27T15:44:57.5347154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3zv_rtn_/_remote_module_non_scriptable.py 2022-09-27T15:44:57.7964810Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:44:57.7975016Z 2022-09-27T15:44:57.7975425Z Running tests... 2022-09-27T15:44:57.7976030Z ---------------------------------------------------------------------- 2022-09-27T15:44:58.0377569Z test_debug_info (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13162 2022-09-27T15:44:58.0403898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13163 2022-09-27T15:44:58.0431721Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13164 2022-09-27T15:44:58.0461928Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13165 2022-09-27T15:44:59.1683163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:59.1683842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:59.2030872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:59.2031571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:59.2138929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:59.2139535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:59.2142903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:59.2143512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:59.2174083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:44:59.2174970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:44:59.2517686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:59.2518788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:59.2525799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:59.2526414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:59.2555408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:44:59.2556060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:44:59.3295104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8s9ga74 2022-09-27T15:44:59.3296228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8s9ga74/_remote_module_non_scriptable.py 2022-09-27T15:44:59.3772284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgx26s9z7 2022-09-27T15:44:59.3773399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgx26s9z7/_remote_module_non_scriptable.py 2022-09-27T15:44:59.3775814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qspnczm 2022-09-27T15:44:59.3778726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qspnczm/_remote_module_non_scriptable.py 2022-09-27T15:44:59.3823129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj457ud0x 2022-09-27T15:44:59.3824637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj457ud0x/_remote_module_non_scriptable.py 2022-09-27T15:44:59.5882423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:44:59.6370225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:44:59.6386584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:44:59.6467538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:00.2518767Z ok (2.454s) 2022-09-27T15:45:00.2518988Z 2022-09-27T15:45:00.2519413Z ---------------------------------------------------------------------- 2022-09-27T15:45:00.2519796Z Ran 1 test in 2.454s 2022-09-27T15:45:00.2519938Z 2022-09-27T15:45:00.2520003Z OK 2022-09-27T15:45:00.2520146Z 2022-09-27T15:45:00.2520266Z Generating XML reports... 2022-09-27T15:45:00.2555252Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154457.xml 2022-09-27T15:45:01.6491348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:01.6491716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:01.6501793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:01.8139958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:01.8140500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2iprnfb 2022-09-27T15:45:01.8141188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2iprnfb/_remote_module_non_scriptable.py 2022-09-27T15:45:02.0852661Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:02.0862357Z 2022-09-27T15:45:02.0862495Z Running tests... 2022-09-27T15:45:02.0863057Z ---------------------------------------------------------------------- 2022-09-27T15:45:02.0874197Z test_default_timeout_used (__main__.TensorPipeRpcTest) 2022-09-27T15:45:02.3374195Z Tests that if no timeout is passed into rpc_async and rpc_sync, then the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13389 2022-09-27T15:45:02.3402337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13390 2022-09-27T15:45:02.3429888Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13391 2022-09-27T15:45:02.3460490Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13392 2022-09-27T15:45:03.5062319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:03.5062921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:03.5358783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:03.5359493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:03.5417559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:03.5418193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:03.5603568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:03.5604181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:03.5649679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:03.5650331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:03.5732723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:03.5733373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:03.5974252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:03.5974917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:03.6019091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:03.6019707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:03.6698950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9j5pt9yh 2022-09-27T15:45:03.6700676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9j5pt9yh/_remote_module_non_scriptable.py 2022-09-27T15:45:03.7002768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72pgg6x5 2022-09-27T15:45:03.7004621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72pgg6x5/_remote_module_non_scriptable.py 2022-09-27T15:45:03.7243187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78fxb4fm 2022-09-27T15:45:03.7244397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78fxb4fm/_remote_module_non_scriptable.py 2022-09-27T15:45:03.7285918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpflzuiue_ 2022-09-27T15:45:03.7287258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpflzuiue_/_remote_module_non_scriptable.py 2022-09-27T15:45:03.9305779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:45:03.9575007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:03.9813699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:45:03.9843060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:45:06.6552167Z ok (4.569s) 2022-09-27T15:45:06.6552496Z 2022-09-27T15:45:06.6553021Z ---------------------------------------------------------------------- 2022-09-27T15:45:06.6553310Z Ran 1 test in 4.569s 2022-09-27T15:45:06.6553433Z 2022-09-27T15:45:06.6553495Z OK 2022-09-27T15:45:06.6553575Z 2022-09-27T15:45:06.6553953Z Generating XML reports... 2022-09-27T15:45:06.6589449Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154502.xml 2022-09-27T15:45:08.0474339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:08.0474898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:08.0483314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:08.0483794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:08.2115917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt1hyzdr 2022-09-27T15:45:08.2117235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt1hyzdr/_remote_module_non_scriptable.py 2022-09-27T15:45:08.4753001Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:08.4762421Z 2022-09-27T15:45:08.4762726Z Running tests... 2022-09-27T15:45:08.4763348Z ---------------------------------------------------------------------- 2022-09-27T15:45:08.7157982Z test_disable_gil_profiling (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13612 2022-09-27T15:45:08.7185555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13613 2022-09-27T15:45:08.7213584Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13614 2022-09-27T15:45:08.7242658Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13615 2022-09-27T15:45:09.8202563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:09.8203238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:09.8394053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:09.8394719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:09.8428995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:09.8429495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:09.8441628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:09.8442129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:09.8573891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:09.8574410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:09.8774516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:09.8775186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:09.8808452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:09.8809113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:09.8815418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:09.8816007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:09.9829651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakr5koyp 2022-09-27T15:45:09.9830599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakr5koyp/_remote_module_non_scriptable.py 2022-09-27T15:45:10.0037806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0wnottb 2022-09-27T15:45:10.0039417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0wnottb/_remote_module_non_scriptable.py 2022-09-27T15:45:10.0072071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2schm7e 2022-09-27T15:45:10.0074647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2schm7e/_remote_module_non_scriptable.py 2022-09-27T15:45:10.0075748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpolul9aeu 2022-09-27T15:45:10.0078842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpolul9aeu/_remote_module_non_scriptable.py 2022-09-27T15:45:10.2385947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:10.2594915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:45:10.2635049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:45:10.2711013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:45:10.9299061Z ok (2.453s) 2022-09-27T15:45:10.9299305Z 2022-09-27T15:45:10.9299840Z ---------------------------------------------------------------------- 2022-09-27T15:45:10.9300283Z Ran 1 test in 2.454s 2022-09-27T15:45:10.9300411Z 2022-09-27T15:45:10.9300463Z OK 2022-09-27T15:45:10.9300556Z 2022-09-27T15:45:10.9300651Z Generating XML reports... 2022-09-27T15:45:10.9336046Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154508.xml 2022-09-27T15:45:12.3395960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:12.3396361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:12.3404012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:12.3404376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:12.4986468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qdfxhno 2022-09-27T15:45:12.4987113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qdfxhno/_remote_module_non_scriptable.py 2022-09-27T15:45:12.7599301Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:12.7609527Z 2022-09-27T15:45:12.7609966Z Running tests... 2022-09-27T15:45:12.7610358Z ---------------------------------------------------------------------- 2022-09-27T15:45:13.0021566Z test_dist_init_decorator (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13835 2022-09-27T15:45:13.0049677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13836 2022-09-27T15:45:13.0077398Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13837 2022-09-27T15:45:13.0106577Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13838 2022-09-27T15:45:14.1037183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:14.1037837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:14.1375023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:14.1375727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:14.1407807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:14.1408486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:14.1748096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:14.1749021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:14.1861876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:14.1862783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:14.2080219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:14.2080846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:14.2232567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:14.2233243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:14.2459918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:14.2460511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:14.2692280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwezgm3s 2022-09-27T15:45:14.2693799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwezgm3s/_remote_module_non_scriptable.py 2022-09-27T15:45:14.3016371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnry_hw75 2022-09-27T15:45:14.3017480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnry_hw75/_remote_module_non_scriptable.py 2022-09-27T15:45:14.3505551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0u27nks7 2022-09-27T15:45:14.3506731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0u27nks7/_remote_module_non_scriptable.py 2022-09-27T15:45:14.3729127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmiw8k16 2022-09-27T15:45:14.3730128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmiw8k16/_remote_module_non_scriptable.py 2022-09-27T15:45:14.5265685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:45:14.5606301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:14.6095138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:45:14.6438180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:45:15.2162629Z ok (2.455s) 2022-09-27T15:45:15.2162797Z 2022-09-27T15:45:15.2163205Z ---------------------------------------------------------------------- 2022-09-27T15:45:15.2163658Z Ran 1 test in 2.455s 2022-09-27T15:45:15.2163861Z 2022-09-27T15:45:15.2163961Z OK 2022-09-27T15:45:15.2164064Z 2022-09-27T15:45:15.2164158Z Generating XML reports... 2022-09-27T15:45:15.2199528Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154512.xml 2022-09-27T15:45:16.5994329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:16.5994983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:16.6003478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:16.6003834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:16.7565048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0sja_4g 2022-09-27T15:45:16.7566120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0sja_4g/_remote_module_non_scriptable.py 2022-09-27T15:45:17.0185566Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:17.0195764Z 2022-09-27T15:45:17.0196172Z Running tests... 2022-09-27T15:45:17.0196586Z ---------------------------------------------------------------------- 2022-09-27T15:45:17.2624904Z test_duplicate_name (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14058 2022-09-27T15:45:17.2653868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14059 2022-09-27T15:45:17.2682297Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14060 2022-09-27T15:45:17.2712842Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14061 2022-09-27T15:45:18.4104690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:18.4105355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:18.4444543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:18.4445163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:18.4452654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:18.4453334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:18.4793316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:18.4793939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:18.4876274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:18.4876891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:18.4959477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:18.4960098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:18.5248775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:18.5249428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:18.5465043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:18.5465669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:18.5724240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjuo_2rb 2022-09-27T15:45:18.5725571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjuo_2rb/_remote_module_non_scriptable.py 2022-09-27T15:45:18.6082921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7jm80lh 2022-09-27T15:45:18.6083719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7jm80lh/_remote_module_non_scriptable.py 2022-09-27T15:45:18.6512474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqnkom7im 2022-09-27T15:45:18.6513229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqnkom7im/_remote_module_non_scriptable.py 2022-09-27T15:45:18.6741495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjs2y0ta_ 2022-09-27T15:45:18.6742288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjs2y0ta_/_remote_module_non_scriptable.py 2022-09-27T15:45:18.8300004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:45:18.8640599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:45:18.9071608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:45:18.9329288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:19.3767263Z ok (2.357s) 2022-09-27T15:45:19.3767487Z 2022-09-27T15:45:19.3768012Z ---------------------------------------------------------------------- 2022-09-27T15:45:19.3768631Z Ran 1 test in 2.357s 2022-09-27T15:45:19.3768755Z 2022-09-27T15:45:19.3768815Z OK 2022-09-27T15:45:19.3768908Z 2022-09-27T15:45:19.3769060Z Generating XML reports... 2022-09-27T15:45:19.3804447Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154517.xml 2022-09-27T15:45:20.7411008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:20.7411448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:20.7419670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:20.7420050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:20.8976195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnw8yowdp 2022-09-27T15:45:20.8977421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnw8yowdp/_remote_module_non_scriptable.py 2022-09-27T15:45:21.1581782Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:21.1590342Z 2022-09-27T15:45:21.1590491Z Running tests... 2022-09-27T15:45:21.1591115Z ---------------------------------------------------------------------- 2022-09-27T15:45:21.4065987Z test_duplicate_name_2 (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14193 2022-09-27T15:45:21.4093736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14194 2022-09-27T15:45:21.4122114Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14195 2022-09-27T15:45:21.4151100Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14196 2022-09-27T15:45:22.5537388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:22.5538010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:22.5621089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:22.5621749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:22.5628825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:22.5629460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:22.5928032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:22.5928686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:22.5938888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:22.5939816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:22.6008755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:22.6009304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:22.6022036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:22.6022596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:22.6314703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:22.6315369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:22.7229882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb05e7k1t 2022-09-27T15:45:22.7230618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb05e7k1t/_remote_module_non_scriptable.py 2022-09-27T15:45:22.7305147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpse626yzr 2022-09-27T15:45:22.7306311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpse626yzr/_remote_module_non_scriptable.py 2022-09-27T15:45:22.7463589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmek_hg7 2022-09-27T15:45:22.7464586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmek_hg7/_remote_module_non_scriptable.py 2022-09-27T15:45:22.7595427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzx4l6rfa 2022-09-27T15:45:22.7596795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzx4l6rfa/_remote_module_non_scriptable.py 2022-09-27T15:45:22.9814391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:45:22.9874718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:45:23.0040240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:45:23.0183801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:23.4204192Z ok (2.261s) 2022-09-27T15:45:23.4204431Z 2022-09-27T15:45:23.4204767Z ---------------------------------------------------------------------- 2022-09-27T15:45:23.4205015Z Ran 1 test in 2.261s 2022-09-27T15:45:23.4205131Z 2022-09-27T15:45:23.4205190Z OK 2022-09-27T15:45:23.4205305Z 2022-09-27T15:45:23.4205433Z Generating XML reports... 2022-09-27T15:45:23.4240642Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154521.xml 2022-09-27T15:45:24.7908653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:24.7909103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:24.7917447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:24.7917797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:24.9484997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppcsf0_tt 2022-09-27T15:45:24.9485911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppcsf0_tt/_remote_module_non_scriptable.py 2022-09-27T15:45:25.2095152Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:25.2105391Z 2022-09-27T15:45:25.2105650Z Running tests... 2022-09-27T15:45:25.2106249Z ---------------------------------------------------------------------- 2022-09-27T15:45:25.4551578Z test_expected_src (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14328 2022-09-27T15:45:25.4579341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14329 2022-09-27T15:45:25.4607270Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14330 2022-09-27T15:45:25.4636495Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14331 2022-09-27T15:45:26.5548278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:26.5548932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:26.5897231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:26.5897899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:26.6048462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:26.6049068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:26.6123038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:26.6123873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:26.6306068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:26.6306746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:26.6420258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:26.6420821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:26.6491981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:26.6492537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:26.6661178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:26.6661832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:26.7172027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptympw70j 2022-09-27T15:45:26.7173388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptympw70j/_remote_module_non_scriptable.py 2022-09-27T15:45:26.7689644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23ik1ltv 2022-09-27T15:45:26.7690545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23ik1ltv/_remote_module_non_scriptable.py 2022-09-27T15:45:26.7762080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplfx8dak6 2022-09-27T15:45:26.7763690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplfx8dak6/_remote_module_non_scriptable.py 2022-09-27T15:45:26.7934160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wiq_fcc 2022-09-27T15:45:26.7935187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wiq_fcc/_remote_module_non_scriptable.py 2022-09-27T15:45:26.9945214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:27.0302513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:45:27.0337787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:45:27.0506657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:45:27.6691904Z ok (2.458s) 2022-09-27T15:45:27.6692153Z 2022-09-27T15:45:27.6692669Z ---------------------------------------------------------------------- 2022-09-27T15:45:27.6693113Z Ran 1 test in 2.459s 2022-09-27T15:45:27.6693320Z 2022-09-27T15:45:27.6693430Z OK 2022-09-27T15:45:27.6693596Z 2022-09-27T15:45:27.6693760Z Generating XML reports... 2022-09-27T15:45:27.6730760Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154525.xml 2022-09-27T15:45:29.0415963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:29.0416362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:29.0424133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:29.0424735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:29.1982371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_w_iem2b 2022-09-27T15:45:29.1983204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_w_iem2b/_remote_module_non_scriptable.py 2022-09-27T15:45:29.4595076Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:29.4604521Z 2022-09-27T15:45:29.4604663Z Running tests... 2022-09-27T15:45:29.4605310Z ---------------------------------------------------------------------- 2022-09-27T15:45:29.7034882Z test_function_not_on_callee (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14551 2022-09-27T15:45:29.7062235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14552 2022-09-27T15:45:29.7090393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14553 2022-09-27T15:45:29.7120049Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14554 2022-09-27T15:45:30.8176676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:30.8177368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:30.8205669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:30.8206337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:30.8538356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:30.8539067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:30.8576772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:30.8577401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:30.8674699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:30.8675376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:30.8902585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:30.8903084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:30.9055760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:30.9056372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:30.9306853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:30.9307532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:30.9821976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xtq1vb1 2022-09-27T15:45:30.9823161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xtq1vb1/_remote_module_non_scriptable.py 2022-09-27T15:45:30.9869115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprae8m35c 2022-09-27T15:45:30.9870161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprae8m35c/_remote_module_non_scriptable.py 2022-09-27T15:45:31.0357900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpag2cbtnv 2022-09-27T15:45:31.0358693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpag2cbtnv/_remote_module_non_scriptable.py 2022-09-27T15:45:31.0671897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxodu89t8 2022-09-27T15:45:31.0673963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxodu89t8/_remote_module_non_scriptable.py 2022-09-27T15:45:31.2375558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:45:31.2460144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:45:31.2908296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:45:31.3218364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:31.5341107Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:45:31.5344107Z AttributeError("Can't get attribute 'foo_add' on Default RPC pickler does not serialize\n function code. Ensure that UDFs are defined on both caller and\n callee modules.") 2022-09-27T15:45:31.5345462Z Traceback (most recent call last): 2022-09-27T15:45:31.5346353Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 159, in deserialize 2022-09-27T15:45:31.5346987Z ret = unpickler.load() 2022-09-27T15:45:31.5348229Z AttributeError: Can't get attribute 'foo_add' on 2022-09-27T15:45:31.5348915Z 2022-09-27T15:45:31.5349211Z The above exception was the direct cause of the following exception: 2022-09-27T15:45:31.5349576Z 2022-09-27T15:45:31.5349780Z Traceback (most recent call last): 2022-09-27T15:45:31.5350605Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 205, in _run_function 2022-09-27T15:45:31.5351213Z raise python_udf 2022-09-27T15:45:31.5352630Z AttributeError: Can't get attribute 'foo_add' on Default RPC pickler does not serialize 2022-09-27T15:45:31.5353651Z function code. Ensure that UDFs are defined on both caller and 2022-09-27T15:45:31.5354129Z callee modules. 2022-09-27T15:45:31.5354368Z 2022-09-27T15:45:31.9176183Z ok (2.457s) 2022-09-27T15:45:31.9176396Z 2022-09-27T15:45:31.9176859Z ---------------------------------------------------------------------- 2022-09-27T15:45:31.9177264Z Ran 1 test in 2.457s 2022-09-27T15:45:31.9177444Z 2022-09-27T15:45:31.9177532Z OK 2022-09-27T15:45:31.9177675Z 2022-09-27T15:45:31.9177823Z Generating XML reports... 2022-09-27T15:45:31.9214026Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154529.xml 2022-09-27T15:45:33.2925423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:33.2925809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:33.2933717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:33.2934229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:33.4535059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqiyzkli 2022-09-27T15:45:33.4535696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqiyzkli/_remote_module_non_scriptable.py 2022-09-27T15:45:33.7192086Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:33.7202781Z 2022-09-27T15:45:33.7203196Z Running tests... 2022-09-27T15:45:33.7203617Z ---------------------------------------------------------------------- 2022-09-27T15:45:33.9624999Z test_future_done (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14774 2022-09-27T15:45:33.9652739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14775 2022-09-27T15:45:33.9680056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14776 2022-09-27T15:45:33.9709586Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14777 2022-09-27T15:45:35.0426286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:35.0426931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:35.0513273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:35.0513980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:35.0806564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:35.0807400Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:35.0890902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:35.0891504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:35.1085449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:35.1086070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:35.1255183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:35.1255804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:35.1441026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:35.1441628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:35.1618346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:35.1618968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:35.2063184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptb3gmdjy 2022-09-27T15:45:35.2063918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptb3gmdjy/_remote_module_non_scriptable.py 2022-09-27T15:45:35.2166181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_ybq7ta 2022-09-27T15:45:35.2167457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_ybq7ta/_remote_module_non_scriptable.py 2022-09-27T15:45:35.2708609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmj7yedh 2022-09-27T15:45:35.2709672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmj7yedh/_remote_module_non_scriptable.py 2022-09-27T15:45:35.2879171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb72pl1vd 2022-09-27T15:45:35.2879915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb72pl1vd/_remote_module_non_scriptable.py 2022-09-27T15:45:35.4617980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:45:35.4743495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:45:35.5274272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:35.5483889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:45:36.1766484Z ok (2.456s) 2022-09-27T15:45:36.1766728Z 2022-09-27T15:45:36.1767364Z ---------------------------------------------------------------------- 2022-09-27T15:45:36.1767791Z Ran 1 test in 2.456s 2022-09-27T15:45:36.1767997Z 2022-09-27T15:45:36.1768101Z OK 2022-09-27T15:45:36.1768271Z 2022-09-27T15:45:36.1768433Z Generating XML reports... 2022-09-27T15:45:36.1803683Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154533.xml 2022-09-27T15:45:37.5697325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:37.5697697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:37.5705261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:37.5705939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:37.7274061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwlui2hn 2022-09-27T15:45:37.7275352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwlui2hn/_remote_module_non_scriptable.py 2022-09-27T15:45:37.9856265Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:37.9865468Z 2022-09-27T15:45:37.9865606Z Running tests... 2022-09-27T15:45:37.9866248Z ---------------------------------------------------------------------- 2022-09-27T15:45:38.2433931Z test_future_done_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14997 2022-09-27T15:45:38.2461976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14998 2022-09-27T15:45:38.2489347Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14999 2022-09-27T15:45:38.2518994Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15000 2022-09-27T15:45:39.3848904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:39.3849557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:39.4230461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:39.4231105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:39.4301024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:39.4301588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:39.4423151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:39.4423774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:39.4633731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:39.4634427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:39.4692305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:39.4692804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:39.4812144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:39.4812666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:39.5008459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:39.5009106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:39.5507612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwt0rx2gn 2022-09-27T15:45:39.5508416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwt0rx2gn/_remote_module_non_scriptable.py 2022-09-27T15:45:39.5966370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsht9n68v 2022-09-27T15:45:39.5967521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsht9n68v/_remote_module_non_scriptable.py 2022-09-27T15:45:39.6090201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgr998qyy 2022-09-27T15:45:39.6091300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgr998qyy/_remote_module_non_scriptable.py 2022-09-27T15:45:39.6281018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcz8sj_p 2022-09-27T15:45:39.6282191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcz8sj_p/_remote_module_non_scriptable.py 2022-09-27T15:45:39.8126484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:39.8570181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:45:39.8707875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:45:39.8890890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:45:40.1062469Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:45:40.1063211Z ValueError('Expected error') 2022-09-27T15:45:40.1063645Z Traceback (most recent call last): 2022-09-27T15:45:40.1064534Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:45:40.1065258Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:45:40.1066215Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:45:40.1066891Z raise ValueError(expected_err) 2022-09-27T15:45:40.1067334Z ValueError: Expected error 2022-09-27T15:45:40.1067610Z 2022-09-27T15:45:40.1249838Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:45:40.1259004Z ValueError('Expected error') 2022-09-27T15:45:40.1259424Z Traceback (most recent call last): 2022-09-27T15:45:40.1260113Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:45:40.1260616Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:45:40.1261273Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:45:40.1261709Z raise ValueError(expected_err) 2022-09-27T15:45:40.1261997Z ValueError: Expected error 2022-09-27T15:45:40.1265690Z 2022-09-27T15:45:40.1270662Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:45:40.1271073Z ValueError('Expected error') 2022-09-27T15:45:40.1271483Z Traceback (most recent call last): 2022-09-27T15:45:40.1272092Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:45:40.1272677Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:45:40.1273477Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:45:40.1274035Z raise ValueError(expected_err) 2022-09-27T15:45:40.1274255Z ValueError: Expected error 2022-09-27T15:45:40.1274367Z 2022-09-27T15:45:40.1351839Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:45:40.1352386Z ValueError('Expected error') 2022-09-27T15:45:40.1352787Z Traceback (most recent call last): 2022-09-27T15:45:40.1353487Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:45:40.1354082Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:45:40.1355060Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:45:40.1355420Z raise ValueError(expected_err) 2022-09-27T15:45:40.1355616Z ValueError: Expected error 2022-09-27T15:45:40.1355748Z 2022-09-27T15:45:40.5576452Z ok (2.571s) 2022-09-27T15:45:40.5576701Z 2022-09-27T15:45:40.5577117Z ---------------------------------------------------------------------- 2022-09-27T15:45:40.5577513Z Ran 1 test in 2.571s 2022-09-27T15:45:40.5577696Z 2022-09-27T15:45:40.5577770Z OK 2022-09-27T15:45:40.5577924Z 2022-09-27T15:45:40.5578072Z Generating XML reports... 2022-09-27T15:45:40.5614097Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154537.xml 2022-09-27T15:45:41.9197787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:41.9198434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:41.9206128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:41.9206595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:42.0789458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjamxyla8 2022-09-27T15:45:42.0789897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjamxyla8/_remote_module_non_scriptable.py 2022-09-27T15:45:42.3411022Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:42.3428278Z 2022-09-27T15:45:42.3428755Z Running tests... 2022-09-27T15:45:42.3429158Z ---------------------------------------------------------------------- 2022-09-27T15:45:42.5905650Z test_future_in_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15220 2022-09-27T15:45:42.5933735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15221 2022-09-27T15:45:42.5961694Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15222 2022-09-27T15:45:42.5991549Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15223 2022-09-27T15:45:43.6918366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:43.6919110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:43.7086211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:43.7086893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:43.7227099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:43.7227661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:43.7250368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:43.7250887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:43.7316994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:43.7317597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:43.7486412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:43.7486976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:43.7618134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:43.7618795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:43.7643211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:43.7643858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:43.8582155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4cstfhx2 2022-09-27T15:45:43.8582989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4cstfhx2/_remote_module_non_scriptable.py 2022-09-27T15:45:43.8750813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rk6wasg 2022-09-27T15:45:43.8752527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rk6wasg/_remote_module_non_scriptable.py 2022-09-27T15:45:43.8883900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rgfskx5 2022-09-27T15:45:43.8885034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rgfskx5/_remote_module_non_scriptable.py 2022-09-27T15:45:43.8918209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1p9c05kf 2022-09-27T15:45:43.8920195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1p9c05kf/_remote_module_non_scriptable.py 2022-09-27T15:45:44.1137168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:45:44.1305889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:44.1426923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:45:44.1491165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:45:44.8048975Z ok (2.462s) 2022-09-27T15:45:44.8049213Z 2022-09-27T15:45:44.8049627Z ---------------------------------------------------------------------- 2022-09-27T15:45:44.8050012Z Ran 1 test in 2.462s 2022-09-27T15:45:44.8050211Z 2022-09-27T15:45:44.8050292Z OK 2022-09-27T15:45:44.8050427Z 2022-09-27T15:45:44.8050582Z Generating XML reports... 2022-09-27T15:45:44.8087142Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154542.xml 2022-09-27T15:45:46.1901109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:46.1901490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:46.1909018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:46.1909368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:46.3471709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53f5hslg 2022-09-27T15:45:46.3472638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53f5hslg/_remote_module_non_scriptable.py 2022-09-27T15:45:46.6109141Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:46.6119066Z 2022-09-27T15:45:46.6119170Z Running tests... 2022-09-27T15:45:46.6119740Z ---------------------------------------------------------------------- 2022-09-27T15:45:46.8551928Z test_future_nested_callback (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15443 2022-09-27T15:45:46.8579100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15444 2022-09-27T15:45:46.8607476Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15445 2022-09-27T15:45:46.8636814Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15446 2022-09-27T15:45:47.9474341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:47.9475152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:47.9806681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:47.9807274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:47.9812083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:47.9812726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:47.9828368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:47.9829069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:47.9850656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:47.9851207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:48.0215753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:48.0216683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:48.0220534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:48.0221478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:48.0230562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:48.0231163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:48.1118857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbr4x2b74 2022-09-27T15:45:48.1120473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbr4x2b74/_remote_module_non_scriptable.py 2022-09-27T15:45:48.1492937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_ayaxj2 2022-09-27T15:45:48.1493641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rycqofg 2022-09-27T15:45:48.1494201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7yl2pxz 2022-09-27T15:45:48.1494794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_ayaxj2/_remote_module_non_scriptable.py 2022-09-27T15:45:48.1495438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rycqofg/_remote_module_non_scriptable.py 2022-09-27T15:45:48.1496119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7yl2pxz/_remote_module_non_scriptable.py 2022-09-27T15:45:48.3679289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:45:48.4047398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:48.4047961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:45:48.4095271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:45:49.0693451Z ok (2.457s) 2022-09-27T15:45:49.0693686Z 2022-09-27T15:45:49.0694189Z ---------------------------------------------------------------------- 2022-09-27T15:45:49.0694622Z Ran 1 test in 2.457s 2022-09-27T15:45:49.0694742Z 2022-09-27T15:45:49.0694803Z OK 2022-09-27T15:45:49.0694893Z 2022-09-27T15:45:49.0694987Z Generating XML reports... 2022-09-27T15:45:49.0729600Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154546.xml 2022-09-27T15:45:50.4355070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:50.4355467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:50.4363366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:50.4363731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:50.5916630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ynh0cd8 2022-09-27T15:45:50.5917507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ynh0cd8/_remote_module_non_scriptable.py 2022-09-27T15:45:50.8525428Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:50.8535715Z 2022-09-27T15:45:50.8535840Z Running tests... 2022-09-27T15:45:50.8536727Z ---------------------------------------------------------------------- 2022-09-27T15:45:51.0885144Z test_future_wait_twice (__main__.TensorPipeRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/69480 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.234s) 2022-09-27T15:45:51.0886040Z 2022-09-27T15:45:51.0886735Z ---------------------------------------------------------------------- 2022-09-27T15:45:51.0886996Z Ran 1 test in 0.235s 2022-09-27T15:45:51.0887108Z 2022-09-27T15:45:51.0887251Z OK (skipped=1) 2022-09-27T15:45:51.0887361Z 2022-09-27T15:45:51.0887433Z Generating XML reports... 2022-09-27T15:45:51.0909254Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154550.xml 2022-09-27T15:45:52.3969868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:52.3970246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:52.3978474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:52.3978815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:52.5528410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgb5yogpn 2022-09-27T15:45:52.5529934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgb5yogpn/_remote_module_non_scriptable.py 2022-09-27T15:45:52.8155003Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:52.8164634Z 2022-09-27T15:45:52.8164762Z Running tests... 2022-09-27T15:45:52.8165224Z ---------------------------------------------------------------------- 2022-09-27T15:45:53.0606250Z test_get_worker_infos (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15676 2022-09-27T15:45:53.0633823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15677 2022-09-27T15:45:53.0661595Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15678 2022-09-27T15:45:53.0690874Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15679 2022-09-27T15:45:54.1530805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:54.1531425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:54.1755334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:54.1756017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:54.1832091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:54.1832735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:54.1833535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:54.1834155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:54.1908904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:54.1909436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:54.2145124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:54.2145775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:54.2211347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:54.2212052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:54.2216685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:54.2217244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:54.3172552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_6aioeu 2022-09-27T15:45:54.3173600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_6aioeu/_remote_module_non_scriptable.py 2022-09-27T15:45:54.3421960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqesg61vr 2022-09-27T15:45:54.3423422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqesg61vr/_remote_module_non_scriptable.py 2022-09-27T15:45:54.3472219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qulh7vs 2022-09-27T15:45:54.3473519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qulh7vs/_remote_module_non_scriptable.py 2022-09-27T15:45:54.3478796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0ojliek 2022-09-27T15:45:54.3480913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0ojliek/_remote_module_non_scriptable.py 2022-09-27T15:45:54.5728357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:45:54.6014492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:45:54.6036820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:54.6052069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:45:55.2747935Z ok (2.458s) 2022-09-27T15:45:55.2748237Z 2022-09-27T15:45:55.2748680Z ---------------------------------------------------------------------- 2022-09-27T15:45:55.2749069Z Ran 1 test in 2.458s 2022-09-27T15:45:55.2749258Z 2022-09-27T15:45:55.2749351Z OK 2022-09-27T15:45:55.2749475Z 2022-09-27T15:45:55.2749617Z Generating XML reports... 2022-09-27T15:45:55.2785316Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154552.xml 2022-09-27T15:45:56.6696590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:56.6696980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:56.6705340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:56.6705714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:56.8299354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpst17wzz3 2022-09-27T15:45:56.8300160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpst17wzz3/_remote_module_non_scriptable.py 2022-09-27T15:45:57.0942652Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:45:57.0952011Z 2022-09-27T15:45:57.0952156Z Running tests... 2022-09-27T15:45:57.0952739Z ---------------------------------------------------------------------- 2022-09-27T15:45:57.0956719Z test_graceful_shutdown_with_uneven_workload (__main__.TensorPipeRpcTest) 2022-09-27T15:45:57.3376266Z Test graceful termination. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15899 2022-09-27T15:45:57.3403846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15900 2022-09-27T15:45:57.3431697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15901 2022-09-27T15:45:57.3461202Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15902 2022-09-27T15:45:58.4849778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:58.4850423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:58.4926561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:58.4927166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:58.5043226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:58.5044099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:58.5131834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:45:58.5132643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:45:58.5236518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:58.5237167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:58.5310662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:58.5311384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:58.5419853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:58.5420480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:58.5504771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:45:58.5505420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:45:58.6499563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpao_ki4sv 2022-09-27T15:45:58.6500355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpao_ki4sv/_remote_module_non_scriptable.py 2022-09-27T15:45:58.6572877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmbrau9cu 2022-09-27T15:45:58.6574368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmbrau9cu/_remote_module_non_scriptable.py 2022-09-27T15:45:58.6667458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv828nrkx 2022-09-27T15:45:58.6668820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv828nrkx/_remote_module_non_scriptable.py 2022-09-27T15:45:58.6772432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6507gkx 2022-09-27T15:45:58.6773756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6507gkx/_remote_module_non_scriptable.py 2022-09-27T15:45:58.9198283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:45:58.9240788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:45:58.9306081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:45:58.9574610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:45:59.7522590Z ok (2.657s) 2022-09-27T15:45:59.7522873Z 2022-09-27T15:45:59.7523335Z ---------------------------------------------------------------------- 2022-09-27T15:45:59.7523572Z Ran 1 test in 2.657s 2022-09-27T15:45:59.7523688Z 2022-09-27T15:45:59.7523764Z OK 2022-09-27T15:45:59.7523872Z 2022-09-27T15:45:59.7523964Z Generating XML reports... 2022-09-27T15:45:59.7558389Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154557.xml 2022-09-27T15:46:01.1281156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:01.1281828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:01.1290235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:01.1290832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:01.2876157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp171lxhmm 2022-09-27T15:46:01.2877121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp171lxhmm/_remote_module_non_scriptable.py 2022-09-27T15:46:01.5560506Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:01.5570607Z 2022-09-27T15:46:01.5571035Z Running tests... 2022-09-27T15:46:01.5571675Z ---------------------------------------------------------------------- 2022-09-27T15:46:01.8066471Z test_handle_send_exceptions (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16122 2022-09-27T15:46:01.8092858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16123 2022-09-27T15:46:01.8121080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16124 2022-09-27T15:46:01.8150494Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16125 2022-09-27T15:46:02.9463822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:02.9464488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:02.9539885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:02.9540556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:02.9843596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:02.9844210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:02.9916189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:02.9916872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:03.0056541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:03.0057150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:03.0076466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:03.0077086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:03.0440510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:03.0441167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:03.0565496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:03.0566173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:03.1135750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuixewinm 2022-09-27T15:46:03.1136549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuixewinm/_remote_module_non_scriptable.py 2022-09-27T15:46:03.1192144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhujiyhc 2022-09-27T15:46:03.1193477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhujiyhc/_remote_module_non_scriptable.py 2022-09-27T15:46:03.1711939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5lx778bm 2022-09-27T15:46:03.1712734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5lx778bm/_remote_module_non_scriptable.py 2022-09-27T15:46:03.1827271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqg6_7w94 2022-09-27T15:46:03.1828015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqg6_7w94/_remote_module_non_scriptable.py 2022-09-27T15:46:03.3678499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:46:03.3744054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:03.4275818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:03.4367917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:03.6844234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:46:03.6943723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:46:03.6944749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:03.6947912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:46:03.6951082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:03.6952114Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:03.6952783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:46:03.6953361Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:03.6959370Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:03.6960332Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:03.6961513Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:46:03.6967168Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:03.6968071Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:03.6968913Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:03.6969839Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #1 to worker2: ECONNREFUSED: connection refused (this error originated at tensorpipe/transport/uv/connection_impl.cc:62) 2022-09-27T15:46:03.6976002Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #2 to worker2: ECONNREFUSED: connection refused (this error originated at tensorpipe/transport/uv/connection_impl.cc:62) 2022-09-27T15:46:04.1208097Z ok (2.563s) 2022-09-27T15:46:04.1208378Z 2022-09-27T15:46:04.1208880Z ---------------------------------------------------------------------- 2022-09-27T15:46:04.1209236Z Ran 1 test in 2.564s 2022-09-27T15:46:04.1209381Z 2022-09-27T15:46:04.1209442Z OK 2022-09-27T15:46:04.1209539Z 2022-09-27T15:46:04.1209646Z Generating XML reports... 2022-09-27T15:46:04.1245700Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154601.xml 2022-09-27T15:46:05.4750280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:05.4750688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:05.4760163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:05.4760529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:05.6385416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujuy8vnv 2022-09-27T15:46:05.6385999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujuy8vnv/_remote_module_non_scriptable.py 2022-09-27T15:46:05.9022500Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:05.9032248Z 2022-09-27T15:46:05.9032360Z Running tests... 2022-09-27T15:46:05.9033229Z ---------------------------------------------------------------------- 2022-09-27T15:46:06.1509152Z test_ignore_rref_leak (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16357 2022-09-27T15:46:06.1537102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16358 2022-09-27T15:46:06.1565397Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16359 2022-09-27T15:46:06.1594157Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16360 2022-09-27T15:46:07.2873568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:07.2874234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:07.3233598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:07.3234319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:07.3243693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:07.3244323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:07.3605160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:07.3605813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:07.3681223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:07.3681840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:07.3789316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:07.3789969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:07.4049574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:07.4050262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:07.4167893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:07.4168531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:07.4526938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6wlq849 2022-09-27T15:46:07.4528315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6wlq849/_remote_module_non_scriptable.py 2022-09-27T15:46:07.4871913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjk1hsdg 2022-09-27T15:46:07.4872738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjk1hsdg/_remote_module_non_scriptable.py 2022-09-27T15:46:07.5321436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4drj9hr 2022-09-27T15:46:07.5322154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4drj9hr/_remote_module_non_scriptable.py 2022-09-27T15:46:07.5434911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4wc7_jk 2022-09-27T15:46:07.5435827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4wc7_jk/_remote_module_non_scriptable.py 2022-09-27T15:46:07.7137052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:07.7467826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:07.7891153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:07.8003868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:46:08.0751840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:46:08.0752770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:46:08.0754006Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:08.0754963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:46:08.0755749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:46:08.0756868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:08.0758015Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:08.0848296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:08.1173473Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-09-27T15:46:08.1174976Z Leaking RRef GloballyUniqueId(created_on=3, local_id=0) with fork Id GloballyUniqueId(created_on=3, local_id=1) 2022-09-27T15:46:08.1175433Z 2022-09-27T15:46:08.1188381Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-09-27T15:46:08.1189587Z Leaking RRef GloballyUniqueId(created_on=0, local_id=0) with fork Id GloballyUniqueId(created_on=0, local_id=1) 2022-09-27T15:46:08.1189899Z 2022-09-27T15:46:08.1236077Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-09-27T15:46:08.1237714Z Leaking RRef GloballyUniqueId(created_on=2, local_id=0) with fork Id GloballyUniqueId(created_on=2, local_id=1) 2022-09-27T15:46:08.1238170Z 2022-09-27T15:46:08.1270976Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-09-27T15:46:08.1272537Z Leaking RRef GloballyUniqueId(created_on=1, local_id=0) with fork Id GloballyUniqueId(created_on=1, local_id=1) 2022-09-27T15:46:08.1272878Z 2022-09-27T15:46:08.5654339Z ok (2.662s) 2022-09-27T15:46:08.5654875Z 2022-09-27T15:46:08.5655177Z ---------------------------------------------------------------------- 2022-09-27T15:46:08.5655440Z Ran 1 test in 2.662s 2022-09-27T15:46:08.5655554Z 2022-09-27T15:46:08.5655615Z OK 2022-09-27T15:46:08.5655693Z 2022-09-27T15:46:08.5655784Z Generating XML reports... 2022-09-27T15:46:08.5691937Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154605.xml 2022-09-27T15:46:09.9610908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:09.9611271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:09.9619961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:09.9620323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:10.1190837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppv7_eg4u 2022-09-27T15:46:10.1191989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppv7_eg4u/_remote_module_non_scriptable.py 2022-09-27T15:46:10.3838707Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:10.3848618Z 2022-09-27T15:46:10.3848720Z Running tests... 2022-09-27T15:46:10.3849403Z ---------------------------------------------------------------------- 2022-09-27T15:46:10.6291267Z test_init_pg_then_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16592 2022-09-27T15:46:10.6318140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16593 2022-09-27T15:46:10.6346678Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16594 2022-09-27T15:46:10.6376912Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16595 2022-09-27T15:46:11.7445806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:11.7446450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:11.7578093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:11.7578803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:11.7680736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:11.7681227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:11.7730371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:11.7731035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:11.7821771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:11.7822415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:11.7963029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:11.7963687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:11.8071269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:11.8071998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:11.8115162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:11.8116133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:11.9086901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqagexnnw 2022-09-27T15:46:11.9087800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqagexnnw/_remote_module_non_scriptable.py 2022-09-27T15:46:11.9245338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13s7pc44 2022-09-27T15:46:11.9246148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13s7pc44/_remote_module_non_scriptable.py 2022-09-27T15:46:11.9354820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5qbbzdq 2022-09-27T15:46:11.9356416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5qbbzdq/_remote_module_non_scriptable.py 2022-09-27T15:46:11.9382803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhyuhkv6 2022-09-27T15:46:11.9384336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhyuhkv6/_remote_module_non_scriptable.py 2022-09-27T15:46:12.1674795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:12.1832739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:12.1945434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:12.1959348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:46:12.2396440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:46:12.2496750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:46:12.2598113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:46:12.2598794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:46:12.2599697Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:12.2600478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:12.2601311Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:12.2602147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:12.9434415Z ok (2.558s) 2022-09-27T15:46:12.9434777Z 2022-09-27T15:46:12.9435231Z ---------------------------------------------------------------------- 2022-09-27T15:46:12.9435617Z Ran 1 test in 2.559s 2022-09-27T15:46:12.9435801Z 2022-09-27T15:46:12.9435901Z OK 2022-09-27T15:46:12.9436054Z 2022-09-27T15:46:12.9436195Z Generating XML reports... 2022-09-27T15:46:12.9471574Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154610.xml 2022-09-27T15:46:14.3018745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:14.3019147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:14.3026777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:14.3027139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:14.4603430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6zvn7jic 2022-09-27T15:46:14.4604065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6zvn7jic/_remote_module_non_scriptable.py 2022-09-27T15:46:14.7207183Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:14.7216698Z 2022-09-27T15:46:14.7216813Z Running tests... 2022-09-27T15:46:14.7217736Z ---------------------------------------------------------------------- 2022-09-27T15:46:14.9710541Z test_init_rpc_then_pg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16827 2022-09-27T15:46:14.9738381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16828 2022-09-27T15:46:14.9766362Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16829 2022-09-27T15:46:14.9796052Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16830 2022-09-27T15:46:16.1331961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:16.1332646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:16.1411095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:16.1411630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:16.1709079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:16.1709718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:16.1784684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:16.1785346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:16.1892841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:16.1893465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:16.2065179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:16.2065765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:16.2247448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:16.2248132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:16.2438199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:16.2438835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:16.2987614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfiswkr_j 2022-09-27T15:46:16.2988358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfiswkr_j/_remote_module_non_scriptable.py 2022-09-27T15:46:16.3049688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpycrvriox 2022-09-27T15:46:16.3050712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpycrvriox/_remote_module_non_scriptable.py 2022-09-27T15:46:16.3510769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tf0w4fb 2022-09-27T15:46:16.3512458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tf0w4fb/_remote_module_non_scriptable.py 2022-09-27T15:46:16.3695166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu4ytrl4p 2022-09-27T15:46:16.3696285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu4ytrl4p/_remote_module_non_scriptable.py 2022-09-27T15:46:16.5570584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:16.5808240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:16.6102949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:16.6266701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:46:16.9171008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:46:16.9171857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:46:16.9271927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:46:16.9275952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:46:16.9277405Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:16.9375669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:16.9376838Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:16.9378012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:17.3855142Z ok (2.663s) 2022-09-27T15:46:17.3855341Z 2022-09-27T15:46:17.3855850Z ---------------------------------------------------------------------- 2022-09-27T15:46:17.3856249Z Ran 1 test in 2.664s 2022-09-27T15:46:17.3856366Z 2022-09-27T15:46:17.3856425Z OK 2022-09-27T15:46:17.3856522Z 2022-09-27T15:46:17.3856612Z Generating XML reports... 2022-09-27T15:46:17.3891284Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154614.xml 2022-09-27T15:46:18.7601556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:18.7602074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:18.7626792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:18.7627202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:18.9179772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmg42sd0d 2022-09-27T15:46:18.9180283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmg42sd0d/_remote_module_non_scriptable.py 2022-09-27T15:46:19.1770990Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:19.1780283Z 2022-09-27T15:46:19.1780368Z Running tests... 2022-09-27T15:46:19.1780789Z ---------------------------------------------------------------------- 2022-09-27T15:46:19.4171647Z test_init_rpc_twice (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17062 2022-09-27T15:46:19.4199376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17063 2022-09-27T15:46:19.4227222Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17064 2022-09-27T15:46:19.4256674Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17065 2022-09-27T15:46:20.5621044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:20.5621686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:20.6000691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:20.6001303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:20.6028320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:20.6028924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:20.6277682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:20.6278226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:20.6302187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:20.6302991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:20.6384020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:20.6384561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:20.6645834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:20.6646620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:20.6668111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:20.6668771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:20.7411564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp108yaxna 2022-09-27T15:46:20.7412303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp108yaxna/_remote_module_non_scriptable.py 2022-09-27T15:46:20.7660744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzg_p9cjq 2022-09-27T15:46:20.7661525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzg_p9cjq/_remote_module_non_scriptable.py 2022-09-27T15:46:20.8001293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphe2cotq3 2022-09-27T15:46:20.8003080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphe2cotq3/_remote_module_non_scriptable.py 2022-09-27T15:46:20.8015937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfht226x7 2022-09-27T15:46:20.8018917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfht226x7/_remote_module_non_scriptable.py 2022-09-27T15:46:21.0049438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:21.0281311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:21.0562833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:46:21.0605292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:21.0908879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:46:21.0909288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:46:21.1008684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:46:21.1009259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:46:21.1010070Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:21.1010754Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:21.1011341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:21.1011898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:21.9317385Z ok (2.753s) 2022-09-27T15:46:21.9317609Z 2022-09-27T15:46:21.9318011Z ---------------------------------------------------------------------- 2022-09-27T15:46:21.9318298Z Ran 1 test in 2.754s 2022-09-27T15:46:21.9318419Z 2022-09-27T15:46:21.9318480Z OK 2022-09-27T15:46:21.9318576Z 2022-09-27T15:46:21.9318677Z Generating XML reports... 2022-09-27T15:46:21.9353224Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154619.xml 2022-09-27T15:46:23.2992923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:23.2993863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:23.3002306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:23.3002905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:23.4595879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxucpvns_ 2022-09-27T15:46:23.4596978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxucpvns_/_remote_module_non_scriptable.py 2022-09-27T15:46:23.7238533Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:23.7248047Z 2022-09-27T15:46:23.7248184Z Running tests... 2022-09-27T15:46:23.7248815Z ---------------------------------------------------------------------- 2022-09-27T15:46:23.9700866Z test_int_callee (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17461 2022-09-27T15:46:23.9728661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17462 2022-09-27T15:46:23.9758381Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17463 2022-09-27T15:46:23.9786339Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17464 2022-09-27T15:46:25.1046506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:25.1047199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:25.1258767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:25.1259507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:25.1379852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:25.1380378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:25.1399439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:25.1400113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:25.1436485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:25.1437192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:25.1650222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:25.1650775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:25.1769662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:25.1770411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:25.1783694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:25.1784329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:25.2699684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79zl8xry 2022-09-27T15:46:25.2700656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79zl8xry/_remote_module_non_scriptable.py 2022-09-27T15:46:25.2919195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6dfht66z 2022-09-27T15:46:25.2920348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6dfht66z/_remote_module_non_scriptable.py 2022-09-27T15:46:25.3032337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe70uhuei 2022-09-27T15:46:25.3033118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe70uhuei/_remote_module_non_scriptable.py 2022-09-27T15:46:25.3046473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaj7x5n_3 2022-09-27T15:46:25.3047927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaj7x5n_3/_remote_module_non_scriptable.py 2022-09-27T15:46:25.5256107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:25.5461997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:25.5600187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:25.5653966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:46:26.1845007Z ok (2.459s) 2022-09-27T15:46:26.1845254Z 2022-09-27T15:46:26.1845804Z ---------------------------------------------------------------------- 2022-09-27T15:46:26.1846086Z Ran 1 test in 2.460s 2022-09-27T15:46:26.1846205Z 2022-09-27T15:46:26.1846272Z OK 2022-09-27T15:46:26.1846363Z 2022-09-27T15:46:26.1846458Z Generating XML reports... 2022-09-27T15:46:26.1880564Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154623.xml 2022-09-27T15:46:27.5591634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:27.5592287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:27.5601384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:27.5601994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:27.7184396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahd7p2fp 2022-09-27T15:46:27.7185066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahd7p2fp/_remote_module_non_scriptable.py 2022-09-27T15:46:27.9774623Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:27.9784278Z 2022-09-27T15:46:27.9784787Z Running tests... 2022-09-27T15:46:27.9785234Z ---------------------------------------------------------------------- 2022-09-27T15:46:28.2197794Z test_invalid_names (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17684 2022-09-27T15:46:28.2224670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17685 2022-09-27T15:46:28.2252240Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17686 2022-09-27T15:46:28.2283412Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17687 2022-09-27T15:46:29.3889731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:29.3890306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:29.3912298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:29.3913010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:29.4261063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:29.4261665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:29.4280078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:29.4280710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:29.4293889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:29.4294752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:29.4353103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:29.4354049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:29.4665394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:29.4666129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:29.4724537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:29.4725179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:29.5541700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvfiaf5_ 2022-09-27T15:46:29.5542528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvfiaf5_/_remote_module_non_scriptable.py 2022-09-27T15:46:29.5550761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrylke_u 2022-09-27T15:46:29.5552909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrylke_u/_remote_module_non_scriptable.py 2022-09-27T15:46:29.5925301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgl24c8td 2022-09-27T15:46:29.5926052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgl24c8td/_remote_module_non_scriptable.py 2022-09-27T15:46:29.5982952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq7c3a_i7 2022-09-27T15:46:29.5985016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq7c3a_i7/_remote_module_non_scriptable.py 2022-09-27T15:46:29.8139868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:29.8140475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:29.8504476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:29.8550077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:46:30.2335728Z ok (2.255s) 2022-09-27T15:46:30.2335933Z 2022-09-27T15:46:30.2336233Z ---------------------------------------------------------------------- 2022-09-27T15:46:30.2336481Z Ran 1 test in 2.255s 2022-09-27T15:46:30.2336656Z 2022-09-27T15:46:30.2336704Z OK 2022-09-27T15:46:30.2336796Z 2022-09-27T15:46:30.2336890Z Generating XML reports... 2022-09-27T15:46:30.2371283Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154627.xml 2022-09-27T15:46:31.6010030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:31.6010413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:31.6017994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:31.6018412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:31.7562665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ecxmabn 2022-09-27T15:46:31.7563667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ecxmabn/_remote_module_non_scriptable.py 2022-09-27T15:46:32.0169913Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:32.0179537Z 2022-09-27T15:46:32.0179622Z Running tests... 2022-09-27T15:46:32.0180392Z ---------------------------------------------------------------------- 2022-09-27T15:46:32.2619598Z test_local_rref_no_fork (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17743 2022-09-27T15:46:32.2646473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17744 2022-09-27T15:46:32.2673820Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17745 2022-09-27T15:46:32.2703725Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17746 2022-09-27T15:46:33.4335794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:33.4336480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:33.4337273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:33.4337903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:33.4706908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:33.4707597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:33.4709076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:33.4709643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:33.5414584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:33.5415203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:33.5567914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:33.5568564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:33.5780540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:33.5781222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:33.5923887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:33.5924515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:33.5979502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4s0qwlv 2022-09-27T15:46:33.5980907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4s0qwlv/_remote_module_non_scriptable.py 2022-09-27T15:46:33.5982746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqrd12rar 2022-09-27T15:46:33.5985266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqrd12rar/_remote_module_non_scriptable.py 2022-09-27T15:46:33.7034852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbouidb1 2022-09-27T15:46:33.7035671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbouidb1/_remote_module_non_scriptable.py 2022-09-27T15:46:33.7181763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22b7cooe 2022-09-27T15:46:33.7183222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22b7cooe/_remote_module_non_scriptable.py 2022-09-27T15:46:33.8544187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:33.8556079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:33.9602434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:33.9745258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:46:34.5761367Z ok (2.558s) 2022-09-27T15:46:34.5761628Z 2022-09-27T15:46:34.5762071Z ---------------------------------------------------------------------- 2022-09-27T15:46:34.5762551Z Ran 1 test in 2.558s 2022-09-27T15:46:34.5762723Z 2022-09-27T15:46:34.5762786Z OK 2022-09-27T15:46:34.5762867Z 2022-09-27T15:46:34.5762964Z Generating XML reports... 2022-09-27T15:46:34.5797969Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154632.xml 2022-09-27T15:46:35.9440910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:35.9441293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:35.9448194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:35.9448540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:36.1007992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwn1trt7p 2022-09-27T15:46:36.1008939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwn1trt7p/_remote_module_non_scriptable.py 2022-09-27T15:46:36.3619444Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:36.3629972Z 2022-09-27T15:46:36.3630423Z Running tests... 2022-09-27T15:46:36.3630826Z ---------------------------------------------------------------------- 2022-09-27T15:46:36.6048605Z test_local_shutdown (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17966 2022-09-27T15:46:36.6075813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17967 2022-09-27T15:46:36.6103106Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17968 2022-09-27T15:46:36.6132727Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17969 2022-09-27T15:46:37.7118743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:37.7119413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:37.7223372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:37.7224053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:37.7248181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:37.7248852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:37.7364707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:37.7365217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:37.7507212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:37.7507847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:37.7619506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:37.7620136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:37.7642374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:37.7643003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:37.7750140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:37.7750812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:37.8774585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmnkhivj4 2022-09-27T15:46:37.8775779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmnkhivj4/_remote_module_non_scriptable.py 2022-09-27T15:46:37.8888538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ndff_hi 2022-09-27T15:46:37.8890187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ndff_hi/_remote_module_non_scriptable.py 2022-09-27T15:46:37.8906819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg5yirvqc 2022-09-27T15:46:37.8908785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg5yirvqc/_remote_module_non_scriptable.py 2022-09-27T15:46:37.9006732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwppt_cpe 2022-09-27T15:46:37.9008322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwppt_cpe/_remote_module_non_scriptable.py 2022-09-27T15:46:38.1332524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:46:38.1445874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:38.1472236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:38.1557387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:38.4019962Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:46:38.4022556Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:46:38.4023920Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:46:38.4025733Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:38.4026940Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:38.4028070Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:38.8190182Z ok (2.456s) 2022-09-27T15:46:38.8190412Z 2022-09-27T15:46:38.8190934Z ---------------------------------------------------------------------- 2022-09-27T15:46:38.8191355Z Ran 1 test in 2.456s 2022-09-27T15:46:38.8191473Z 2022-09-27T15:46:38.8191538Z OK 2022-09-27T15:46:38.8191630Z 2022-09-27T15:46:38.8191714Z Generating XML reports... 2022-09-27T15:46:38.8229216Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154636.xml 2022-09-27T15:46:40.1956396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:40.1956765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:40.1964909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:40.1965273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:40.3574430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9a900b3 2022-09-27T15:46:40.3575163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9a900b3/_remote_module_non_scriptable.py 2022-09-27T15:46:40.6190067Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:40.6200174Z 2022-09-27T15:46:40.6200294Z Running tests... 2022-09-27T15:46:40.6200744Z ---------------------------------------------------------------------- 2022-09-27T15:46:40.8586013Z test_local_shutdown_with_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18189 2022-09-27T15:46:40.8614264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18190 2022-09-27T15:46:40.8642024Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18191 2022-09-27T15:46:40.8671945Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18192 2022-09-27T15:46:41.9647086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:41.9647753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:41.9830134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:41.9830788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:41.9831635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:41.9832238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:42.0036792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:42.0037332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:42.0059187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:42.0059688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:42.0218559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:42.0219277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:42.0223330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:42.0223884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:42.0422023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:42.0422659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:42.1296488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbc35egs 2022-09-27T15:46:42.1297295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbc35egs/_remote_module_non_scriptable.py 2022-09-27T15:46:42.1474885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6tax0vg 2022-09-27T15:46:42.1476078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6tax0vg/_remote_module_non_scriptable.py 2022-09-27T15:46:42.1491195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_9hx5_k 2022-09-27T15:46:42.1492419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_9hx5_k/_remote_module_non_scriptable.py 2022-09-27T15:46:42.1670699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7np8g79y 2022-09-27T15:46:42.1672116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7np8g79y/_remote_module_non_scriptable.py 2022-09-27T15:46:42.3851060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:42.4105862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:46:42.4233740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:42.4237555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:42.7282286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:46:42.7283147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:46:42.7382878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:46:42.7384314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:46:42.7385883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:42.7387245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:42.7388572Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:42.7389913Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:46:42.7494540Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:46:42.7495449Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:42.7496282Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:42.7497096Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:42.7497910Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:42.7498731Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:42.7499577Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:42.7678936Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:46:43.1730456Z ok (2.553s) 2022-09-27T15:46:43.1730683Z 2022-09-27T15:46:43.1731285Z ---------------------------------------------------------------------- 2022-09-27T15:46:43.1731722Z Ran 1 test in 2.553s 2022-09-27T15:46:43.1731852Z 2022-09-27T15:46:43.1731913Z OK 2022-09-27T15:46:43.1732008Z 2022-09-27T15:46:43.1732099Z Generating XML reports... 2022-09-27T15:46:43.1766839Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154640.xml 2022-09-27T15:46:44.5522180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:44.5522566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:44.5530931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:44.5531279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:44.7103533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzztvy9a 2022-09-27T15:46:44.7104474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzztvy9a/_remote_module_non_scriptable.py 2022-09-27T15:46:44.9719749Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:44.9729347Z 2022-09-27T15:46:44.9729476Z Running tests... 2022-09-27T15:46:44.9730084Z ---------------------------------------------------------------------- 2022-09-27T15:46:45.2127395Z test_local_value_not_on_owner (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18424 2022-09-27T15:46:45.2156250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18425 2022-09-27T15:46:45.2183816Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18426 2022-09-27T15:46:45.2212775Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18427 2022-09-27T15:46:46.3182458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:46.3183139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:46.3527431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:46.3528025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:46.3626930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:46.3627552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:46.3658195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:46.3658770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:46.3659536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:46.3660118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:46.4001131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:46.4001876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:46.4030814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:46.4031550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:46.4036984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:46.4037567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:46.4797212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gqgrrh4 2022-09-27T15:46:46.4798019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gqgrrh4/_remote_module_non_scriptable.py 2022-09-27T15:46:46.5281981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6grivso 2022-09-27T15:46:46.5282636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6grivso/_remote_module_non_scriptable.py 2022-09-27T15:46:46.5320629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5w0jt4m 2022-09-27T15:46:46.5322202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5w0jt4m/_remote_module_non_scriptable.py 2022-09-27T15:46:46.5480894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3x0y0253 2022-09-27T15:46:46.5481677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3x0y0253/_remote_module_non_scriptable.py 2022-09-27T15:46:46.7368647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:46.7827733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:46:46.7890492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:46.8060074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:47.4269052Z ok (2.453s) 2022-09-27T15:46:47.4269331Z 2022-09-27T15:46:47.4269815Z ---------------------------------------------------------------------- 2022-09-27T15:46:47.4270053Z Ran 1 test in 2.454s 2022-09-27T15:46:47.4270400Z 2022-09-27T15:46:47.4270464Z OK 2022-09-27T15:46:47.4270559Z 2022-09-27T15:46:47.4270660Z Generating XML reports... 2022-09-27T15:46:47.4304730Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154644.xml 2022-09-27T15:46:48.7979252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:48.7979638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:48.7988175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:48.7988578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:48.9572777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplfpvm_nu 2022-09-27T15:46:48.9573891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplfpvm_nu/_remote_module_non_scriptable.py 2022-09-27T15:46:49.2175665Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:49.2185331Z 2022-09-27T15:46:49.2185462Z Running tests... 2022-09-27T15:46:49.2186485Z ---------------------------------------------------------------------- 2022-09-27T15:46:49.4615159Z test_mark_future_twice (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18647 2022-09-27T15:46:49.4642460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18648 2022-09-27T15:46:49.4670223Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18649 2022-09-27T15:46:49.4699224Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18650 2022-09-27T15:46:50.5839799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:50.5840473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:50.5986540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:50.5987205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:50.6127554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:50.6128324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:50.6245404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:50.6246060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:50.6395039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:50.6395703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:50.6525198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:50.6525787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:50.6526539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:50.6527096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:50.6890483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:50.6891131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:50.7522579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm902flhh 2022-09-27T15:46:50.7523369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm902flhh/_remote_module_non_scriptable.py 2022-09-27T15:46:50.7662605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdgbkaef 2022-09-27T15:46:50.7663416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdgbkaef/_remote_module_non_scriptable.py 2022-09-27T15:46:50.7805633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05i_bw9q 2022-09-27T15:46:50.7806518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05i_bw9q/_remote_module_non_scriptable.py 2022-09-27T15:46:50.8156795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxar30oi0 2022-09-27T15:46:50.8157637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxar30oi0/_remote_module_non_scriptable.py 2022-09-27T15:46:51.0089088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:51.0234420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:51.0399220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:51.0736292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:46:51.6756273Z ok (2.457s) 2022-09-27T15:46:51.6756506Z 2022-09-27T15:46:51.6756841Z ---------------------------------------------------------------------- 2022-09-27T15:46:51.6757081Z Ran 1 test in 2.457s 2022-09-27T15:46:51.6757199Z 2022-09-27T15:46:51.6757260Z OK 2022-09-27T15:46:51.6757351Z 2022-09-27T15:46:51.6757449Z Generating XML reports... 2022-09-27T15:46:51.6791920Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154649.xml 2022-09-27T15:46:53.0547359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:53.0548006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:53.0557379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:53.0557958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:53.2124186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_rfy0dz 2022-09-27T15:46:53.2125149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_rfy0dz/_remote_module_non_scriptable.py 2022-09-27T15:46:53.4735144Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:53.4745294Z 2022-09-27T15:46:53.4745607Z Running tests... 2022-09-27T15:46:53.4746229Z ---------------------------------------------------------------------- 2022-09-27T15:46:53.7191945Z test_multi_builtin_remote_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18870 2022-09-27T15:46:53.7219029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18871 2022-09-27T15:46:53.7247563Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18872 2022-09-27T15:46:53.7276614Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18873 2022-09-27T15:46:54.8025846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:54.8026497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:54.8246237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:54.8246796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:54.8384265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:54.8384883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:54.8385948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:54.8386545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:54.8438893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:54.8439540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:54.8625927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:54.8626518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:54.8760019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:54.8760658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:54.8805549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:54.8806189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:54.9647872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2hwtun_g 2022-09-27T15:46:54.9648821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2hwtun_g/_remote_module_non_scriptable.py 2022-09-27T15:46:54.9879240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4krmwcw 2022-09-27T15:46:54.9880428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4krmwcw/_remote_module_non_scriptable.py 2022-09-27T15:46:55.0022718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4flwq1xh 2022-09-27T15:46:55.0023901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4flwq1xh/_remote_module_non_scriptable.py 2022-09-27T15:46:55.0053830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44ig1ybq 2022-09-27T15:46:55.0055227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44ig1ybq/_remote_module_non_scriptable.py 2022-09-27T15:46:55.2196439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:55.2455578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:55.2595334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:55.2612135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:46:55.9334281Z ok (2.458s) 2022-09-27T15:46:55.9334569Z 2022-09-27T15:46:55.9335063Z ---------------------------------------------------------------------- 2022-09-27T15:46:55.9335302Z Ran 1 test in 2.459s 2022-09-27T15:46:55.9335426Z 2022-09-27T15:46:55.9335488Z OK 2022-09-27T15:46:55.9335580Z 2022-09-27T15:46:55.9335675Z Generating XML reports... 2022-09-27T15:46:55.9370834Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154653.xml 2022-09-27T15:46:57.3002792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:57.3003483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:57.3012351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:57.3012746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:57.4596651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4b052thc 2022-09-27T15:46:57.4597524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4b052thc/_remote_module_non_scriptable.py 2022-09-27T15:46:57.7200644Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:46:57.7210763Z 2022-09-27T15:46:57.7211015Z Running tests... 2022-09-27T15:46:57.7211694Z ---------------------------------------------------------------------- 2022-09-27T15:46:57.9676180Z test_multi_layer_nested_async_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19093 2022-09-27T15:46:57.9703669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19094 2022-09-27T15:46:57.9731360Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19095 2022-09-27T15:46:57.9761232Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19096 2022-09-27T15:46:59.1102377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:59.1103019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:59.1277957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:59.1278704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:59.1474748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:59.1475324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:59.1639679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:59.1640335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:59.2020879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:59.2021498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:59.2244963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:46:59.2245639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:46:59.2412554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:59.2413179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:59.2628436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:46:59.2629049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:46:59.2761924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelj_7lr0 2022-09-27T15:46:59.2762841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelj_7lr0/_remote_module_non_scriptable.py 2022-09-27T15:46:59.2927850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrx1ei6o 2022-09-27T15:46:59.2928642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrx1ei6o/_remote_module_non_scriptable.py 2022-09-27T15:46:59.3677284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurvpxs9d 2022-09-27T15:46:59.3678717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurvpxs9d/_remote_module_non_scriptable.py 2022-09-27T15:46:59.3883798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6luiouf 2022-09-27T15:46:59.3884948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6luiouf/_remote_module_non_scriptable.py 2022-09-27T15:46:59.5356889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:46:59.5512296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:46:59.6243371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:46:59.6451442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:00.3820605Z ok (2.661s) 2022-09-27T15:47:00.3820872Z 2022-09-27T15:47:00.3821457Z ---------------------------------------------------------------------- 2022-09-27T15:47:00.3821704Z Ran 1 test in 2.661s 2022-09-27T15:47:00.3821902Z 2022-09-27T15:47:00.3821963Z OK 2022-09-27T15:47:00.3822054Z 2022-09-27T15:47:00.3822151Z Generating XML reports... 2022-09-27T15:47:00.3856505Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154657.xml 2022-09-27T15:47:01.7675455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:01.7675849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:01.7683941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:01.7684422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:01.9272546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoy1i0j14 2022-09-27T15:47:01.9273734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoy1i0j14/_remote_module_non_scriptable.py 2022-09-27T15:47:02.1908275Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:02.1918486Z 2022-09-27T15:47:02.1918621Z Running tests... 2022-09-27T15:47:02.1919052Z ---------------------------------------------------------------------- 2022-09-27T15:47:02.4346315Z test_multi_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19316 2022-09-27T15:47:02.4374462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19317 2022-09-27T15:47:02.4403748Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19318 2022-09-27T15:47:02.4434839Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19319 2022-09-27T15:47:03.6219568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:03.6220247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:03.6274885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:03.6275510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:03.6314851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:03.6315533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:03.6433716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:03.6434226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:03.6616421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:03.6617127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:03.6672056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:03.6672787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:03.6712481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:03.6713142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:03.6815578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:03.6816224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:03.7884511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3nd7wl_ 2022-09-27T15:47:03.7885497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3nd7wl_/_remote_module_non_scriptable.py 2022-09-27T15:47:03.7933140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyyra20pp 2022-09-27T15:47:03.7934394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyyra20pp/_remote_module_non_scriptable.py 2022-09-27T15:47:03.7985622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpunynh010 2022-09-27T15:47:03.7987554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpunynh010/_remote_module_non_scriptable.py 2022-09-27T15:47:03.8093200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0hrc7vp_ 2022-09-27T15:47:03.8094173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0hrc7vp_/_remote_module_non_scriptable.py 2022-09-27T15:47:04.0447373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:47:04.0493438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:47:04.0578327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:04.0801797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:47:04.7494981Z ok (2.557s) 2022-09-27T15:47:04.7495261Z 2022-09-27T15:47:04.7495671Z ---------------------------------------------------------------------- 2022-09-27T15:47:04.7495936Z Ran 1 test in 2.558s 2022-09-27T15:47:04.7496041Z 2022-09-27T15:47:04.7496100Z OK 2022-09-27T15:47:04.7496193Z 2022-09-27T15:47:04.7496288Z Generating XML reports... 2022-09-27T15:47:04.7533115Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154702.xml 2022-09-27T15:47:06.0993823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:06.0994225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:06.1002900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:06.1003310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:06.2647780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg6nxmica 2022-09-27T15:47:06.2648816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg6nxmica/_remote_module_non_scriptable.py 2022-09-27T15:47:06.5309043Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:06.5319294Z 2022-09-27T15:47:06.5319382Z Running tests... 2022-09-27T15:47:06.5319945Z ---------------------------------------------------------------------- 2022-09-27T15:47:06.7820431Z test_multi_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19539 2022-09-27T15:47:06.7848449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19540 2022-09-27T15:47:06.7877128Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19541 2022-09-27T15:47:06.7906854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19542 2022-09-27T15:47:07.9013206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:07.9013853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:07.9344009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:07.9344602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:07.9360679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:07.9361622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:07.9394734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:07.9395607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:07.9445320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:07.9445965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:07.9715128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:07.9715850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:07.9768194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:07.9768951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:07.9812479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:07.9813130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:08.0616441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqfyq3kq3 2022-09-27T15:47:08.0630814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqfyq3kq3/_remote_module_non_scriptable.py 2022-09-27T15:47:08.0977426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxs1yiuys 2022-09-27T15:47:08.0978305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxs1yiuys/_remote_module_non_scriptable.py 2022-09-27T15:47:08.1035837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7kfp00t 2022-09-27T15:47:08.1037460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7kfp00t/_remote_module_non_scriptable.py 2022-09-27T15:47:08.1062049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwixzui0d 2022-09-27T15:47:08.1064084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwixzui0d/_remote_module_non_scriptable.py 2022-09-27T15:47:08.3252067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:08.3576333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:47:08.3642067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:47:08.3818996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:47:08.9961931Z ok (2.464s) 2022-09-27T15:47:08.9962101Z 2022-09-27T15:47:08.9962422Z ---------------------------------------------------------------------- 2022-09-27T15:47:08.9962676Z Ran 1 test in 2.464s 2022-09-27T15:47:08.9962826Z 2022-09-27T15:47:08.9962883Z OK 2022-09-27T15:47:08.9962977Z 2022-09-27T15:47:08.9963088Z Generating XML reports... 2022-09-27T15:47:08.9998077Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154706.xml 2022-09-27T15:47:10.3999897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:10.4000541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:10.4008901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:10.4009560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:10.5616405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwq4dxay4 2022-09-27T15:47:10.5617401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwq4dxay4/_remote_module_non_scriptable.py 2022-09-27T15:47:10.8235279Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:10.8245389Z 2022-09-27T15:47:10.8245708Z Running tests... 2022-09-27T15:47:10.8246336Z ---------------------------------------------------------------------- 2022-09-27T15:47:11.0721442Z test_my_parameter_server (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19762 2022-09-27T15:47:11.0749217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19763 2022-09-27T15:47:11.0777775Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19764 2022-09-27T15:47:11.0806399Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19765 2022-09-27T15:47:12.2131371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:12.2132017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:12.2518474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:12.2519166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:12.2883956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:12.2884574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:12.3167504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:12.3168169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:12.3260577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:12.3261192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:12.3261943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:12.3262604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:12.3567243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:12.3567814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:12.3697034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:12.3697677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:12.3822858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxjs_pin 2022-09-27T15:47:12.3823590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxjs_pin/_remote_module_non_scriptable.py 2022-09-27T15:47:12.4533706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxty09qf7 2022-09-27T15:47:12.4535275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxty09qf7/_remote_module_non_scriptable.py 2022-09-27T15:47:12.4840031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpldlujzjx 2022-09-27T15:47:12.4841011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpldlujzjx/_remote_module_non_scriptable.py 2022-09-27T15:47:12.4953839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcrj0e8v 2022-09-27T15:47:12.4954711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcrj0e8v/_remote_module_non_scriptable.py 2022-09-27T15:47:12.6404902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:47:12.7102283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:47:12.7423692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:47:12.7525895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:13.4866490Z ok (2.662s) 2022-09-27T15:47:13.4866721Z 2022-09-27T15:47:13.4867239Z ---------------------------------------------------------------------- 2022-09-27T15:47:13.4867831Z Ran 1 test in 2.662s 2022-09-27T15:47:13.4867950Z 2022-09-27T15:47:13.4868014Z OK 2022-09-27T15:47:13.4868108Z 2022-09-27T15:47:13.4868207Z Generating XML reports... 2022-09-27T15:47:13.4903523Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154710.xml 2022-09-27T15:47:14.8630572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:14.8630962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:14.8639045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:14.8639413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:15.0197353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzsx028qj 2022-09-27T15:47:15.0198205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzsx028qj/_remote_module_non_scriptable.py 2022-09-27T15:47:15.2807980Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:15.2817790Z 2022-09-27T15:47:15.2817913Z Running tests... 2022-09-27T15:47:15.2818318Z ---------------------------------------------------------------------- 2022-09-27T15:47:15.5297543Z test_nested_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20021 2022-09-27T15:47:15.5324506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20022 2022-09-27T15:47:15.5352315Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20023 2022-09-27T15:47:15.5382533Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20024 2022-09-27T15:47:16.6076628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:16.6077236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:16.6211081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:16.6211683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:16.6329972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:16.6330672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:16.6389848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:16.6390347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:16.6448010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:16.6448645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:16.6591844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:16.6592452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:16.6701099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:16.6701751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:16.6761576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:16.6762201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:16.7705382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkb40s6rr 2022-09-27T15:47:16.7706220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkb40s6rr/_remote_module_non_scriptable.py 2022-09-27T15:47:16.7859432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxm5xhzx2 2022-09-27T15:47:16.7860638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxm5xhzx2/_remote_module_non_scriptable.py 2022-09-27T15:47:16.7963036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97fx97s8 2022-09-27T15:47:16.7964424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97fx97s8/_remote_module_non_scriptable.py 2022-09-27T15:47:16.8006501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvo0tw1pn 2022-09-27T15:47:16.8007783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvo0tw1pn/_remote_module_non_scriptable.py 2022-09-27T15:47:17.0265670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:47:17.0464033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:47:17.0528030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:47:17.0566094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:17.7437679Z ok (2.462s) 2022-09-27T15:47:17.7437894Z 2022-09-27T15:47:17.7438255Z ---------------------------------------------------------------------- 2022-09-27T15:47:17.7438494Z Ran 1 test in 2.462s 2022-09-27T15:47:17.7438611Z 2022-09-27T15:47:17.7438675Z OK 2022-09-27T15:47:17.7438773Z 2022-09-27T15:47:17.7438922Z Generating XML reports... 2022-09-27T15:47:17.7473124Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154715.xml 2022-09-27T15:47:19.1152534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:19.1152929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:19.1160990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:19.1161489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:19.2748972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvegvhwy 2022-09-27T15:47:19.2750007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvegvhwy/_remote_module_non_scriptable.py 2022-09-27T15:47:19.5339741Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:19.5349258Z 2022-09-27T15:47:19.5349365Z Running tests... 2022-09-27T15:47:19.5350471Z ---------------------------------------------------------------------- 2022-09-27T15:47:19.7853480Z test_nested_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20244 2022-09-27T15:47:19.7880421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20245 2022-09-27T15:47:19.7908575Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20246 2022-09-27T15:47:19.7939995Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20247 2022-09-27T15:47:20.9370366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:20.9371057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:20.9378691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:20.9379348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:20.9740379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:20.9741017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:20.9746380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:20.9747030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:21.0284058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:21.0284720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:21.0456951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:21.0457602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:21.0665415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:21.0666103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:21.0820914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:21.0821609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:21.1007119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6b73nl7e 2022-09-27T15:47:21.1007935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6b73nl7e/_remote_module_non_scriptable.py 2022-09-27T15:47:21.1019079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rcgfbdg 2022-09-27T15:47:21.1021059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rcgfbdg/_remote_module_non_scriptable.py 2022-09-27T15:47:21.1923145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgtku7pww 2022-09-27T15:47:21.1924130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgtku7pww/_remote_module_non_scriptable.py 2022-09-27T15:47:21.2085939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprvouv6uc 2022-09-27T15:47:21.2087247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprvouv6uc/_remote_module_non_scriptable.py 2022-09-27T15:47:21.3587397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:21.3627397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:47:21.4498441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:47:21.4664481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:47:22.1998832Z ok (2.665s) 2022-09-27T15:47:22.1999001Z 2022-09-27T15:47:22.1999396Z ---------------------------------------------------------------------- 2022-09-27T15:47:22.1999663Z Ran 1 test in 2.665s 2022-09-27T15:47:22.1999766Z 2022-09-27T15:47:22.1999841Z OK 2022-09-27T15:47:22.1999934Z 2022-09-27T15:47:22.2000029Z Generating XML reports... 2022-09-27T15:47:22.2035171Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154719.xml 2022-09-27T15:47:23.6053738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:23.6054395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:23.6062760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:23.6063357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:23.7683692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsy17m8nk 2022-09-27T15:47:23.7684476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsy17m8nk/_remote_module_non_scriptable.py 2022-09-27T15:47:24.0327779Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:24.0337002Z 2022-09-27T15:47:24.0337127Z Running tests... 2022-09-27T15:47:24.0337744Z ---------------------------------------------------------------------- 2022-09-27T15:47:24.2782338Z test_nested_rref (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20467 2022-09-27T15:47:24.2810166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20468 2022-09-27T15:47:24.2838159Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20469 2022-09-27T15:47:24.2867691Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20470 2022-09-27T15:47:25.3875797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:25.3876489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:25.4051372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:25.4051937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:25.4220209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:25.4220939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:25.4257315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:25.4257968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:25.4436695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:25.4437272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:25.4598347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:25.4598986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:25.4610931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:25.4611522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:25.4979871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:25.4980568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:25.5547980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphffs6fz2 2022-09-27T15:47:25.5548799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphffs6fz2/_remote_module_non_scriptable.py 2022-09-27T15:47:25.5708804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa38kgx7b 2022-09-27T15:47:25.5709551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa38kgx7b/_remote_module_non_scriptable.py 2022-09-27T15:47:25.5865858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gmu4hzl 2022-09-27T15:47:25.5867221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gmu4hzl/_remote_module_non_scriptable.py 2022-09-27T15:47:25.6237949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9kmr63c 2022-09-27T15:47:25.6239044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9kmr63c/_remote_module_non_scriptable.py 2022-09-27T15:47:25.8145759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:47:25.8273603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:25.8425542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:47:25.8823188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:47:26.4922395Z ok (2.458s) 2022-09-27T15:47:26.4922649Z 2022-09-27T15:47:26.4923103Z ---------------------------------------------------------------------- 2022-09-27T15:47:26.4923499Z Ran 1 test in 2.458s 2022-09-27T15:47:26.4923688Z 2022-09-27T15:47:26.4923784Z OK 2022-09-27T15:47:26.4923925Z 2022-09-27T15:47:26.4924054Z Generating XML reports... 2022-09-27T15:47:26.4958902Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154724.xml 2022-09-27T15:47:27.8617837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:27.8618207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:27.8626384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:27.8626945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:28.0256661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1r45eqls 2022-09-27T15:47:28.0257349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1r45eqls/_remote_module_non_scriptable.py 2022-09-27T15:47:28.2894550Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:28.2905794Z 2022-09-27T15:47:28.2906084Z Running tests... 2022-09-27T15:47:28.2906769Z ---------------------------------------------------------------------- 2022-09-27T15:47:28.5352086Z test_nested_rref_stress (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20690 2022-09-27T15:47:28.5379585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20691 2022-09-27T15:47:28.5406631Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20692 2022-09-27T15:47:28.5436171Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20693 2022-09-27T15:47:29.6450179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:29.6450837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:29.6637609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:29.6638102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:29.6833882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:29.6834726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:29.6835571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:29.6836251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:29.7020062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:29.7020635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:29.7034689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:29.7035183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:29.7211319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:29.7211936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:29.7407125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:29.7407985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:29.8107516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0q9skku 2022-09-27T15:47:29.8108405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0q9skku/_remote_module_non_scriptable.py 2022-09-27T15:47:29.8291511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvng923b 2022-09-27T15:47:29.8292304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvng923b/_remote_module_non_scriptable.py 2022-09-27T15:47:29.8491331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvwl62g67 2022-09-27T15:47:29.8492388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvwl62g67/_remote_module_non_scriptable.py 2022-09-27T15:47:29.8680222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vv06123 2022-09-27T15:47:29.8682037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vv06123/_remote_module_non_scriptable.py 2022-09-27T15:47:30.0700781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:47:30.0870785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:47:30.1264253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:47:30.1267628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:30.8495170Z ok (2.559s) 2022-09-27T15:47:30.8495433Z 2022-09-27T15:47:30.8495772Z ---------------------------------------------------------------------- 2022-09-27T15:47:30.8496145Z Ran 1 test in 2.559s 2022-09-27T15:47:30.8496349Z 2022-09-27T15:47:30.8496440Z OK 2022-09-27T15:47:30.8496557Z 2022-09-27T15:47:30.8496655Z Generating XML reports... 2022-09-27T15:47:30.8534051Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154728.xml 2022-09-27T15:47:32.2317177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:32.2317584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:32.2325019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:32.2325366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:32.3921590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1dkhj8f 2022-09-27T15:47:32.3923022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1dkhj8f/_remote_module_non_scriptable.py 2022-09-27T15:47:32.6536805Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:32.6546729Z 2022-09-27T15:47:32.6547067Z Running tests... 2022-09-27T15:47:32.6547724Z ---------------------------------------------------------------------- 2022-09-27T15:47:32.8972400Z test_non_cont_tensors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20913 2022-09-27T15:47:32.8999710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20914 2022-09-27T15:47:32.9027052Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20915 2022-09-27T15:47:32.9056687Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20916 2022-09-27T15:47:33.9854051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:33.9854673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:34.0004459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:34.0005081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:34.0223818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:34.0224606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:34.0306685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:34.0307296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:34.0375507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:34.0376140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:34.0470064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:34.0470664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:34.0670405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:34.0671058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:34.0846393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:34.0847079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:34.1684162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpso2ivm5_ 2022-09-27T15:47:34.1684860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3cy8dz5k 2022-09-27T15:47:34.1685518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpso2ivm5_/_remote_module_non_scriptable.py 2022-09-27T15:47:34.1686191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3cy8dz5k/_remote_module_non_scriptable.py 2022-09-27T15:47:34.1977093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvll3wfie 2022-09-27T15:47:34.1977835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvll3wfie/_remote_module_non_scriptable.py 2022-09-27T15:47:34.2130168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1oydckd 2022-09-27T15:47:34.2130989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1oydckd/_remote_module_non_scriptable.py 2022-09-27T15:47:34.4240122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:47:34.4242150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:47:34.4545168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:47:34.4717309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:35.1116485Z ok (2.457s) 2022-09-27T15:47:35.1116870Z 2022-09-27T15:47:35.1117323Z ---------------------------------------------------------------------- 2022-09-27T15:47:35.1117681Z Ran 1 test in 2.457s 2022-09-27T15:47:35.1117801Z 2022-09-27T15:47:35.1117900Z OK 2022-09-27T15:47:35.1118030Z 2022-09-27T15:47:35.1118113Z Generating XML reports... 2022-09-27T15:47:35.1176607Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154732.xml 2022-09-27T15:47:36.4798899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:36.4799280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:36.4806982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:36.4807344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:36.6379786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8j1o2x1 2022-09-27T15:47:36.6380662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8j1o2x1/_remote_module_non_scriptable.py 2022-09-27T15:47:36.9022087Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:36.9032545Z 2022-09-27T15:47:36.9033015Z Running tests... 2022-09-27T15:47:36.9033461Z ---------------------------------------------------------------------- 2022-09-27T15:47:37.1488104Z test_non_garbage_collected_user_rref_due_to_local_circular_dependency (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21136 2022-09-27T15:47:37.1515412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21137 2022-09-27T15:47:37.1547589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21138 2022-09-27T15:47:37.1578826Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21139 2022-09-27T15:47:38.1959661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:38.1960280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:38.2316721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:38.2317318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:38.2324931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:38.2325576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:38.2328477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:38.2329140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:38.2689545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:38.2690224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:38.2698467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:38.2699045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:38.3032029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:38.3032727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:38.3375028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:38.3375610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:38.3591507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpag63j9_o 2022-09-27T15:47:38.3592552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpag63j9_o/_remote_module_non_scriptable.py 2022-09-27T15:47:38.3959999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_schwnt3 2022-09-27T15:47:38.3960707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmjmby9c 2022-09-27T15:47:38.3961374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_schwnt3/_remote_module_non_scriptable.py 2022-09-27T15:47:38.3962032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmjmby9c/_remote_module_non_scriptable.py 2022-09-27T15:47:38.4631497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1om5os9 2022-09-27T15:47:38.4632649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1om5os9/_remote_module_non_scriptable.py 2022-09-27T15:47:38.6131641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:47:38.6505060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:47:38.6512200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:47:38.7173388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:39.3634236Z ok (2.460s) 2022-09-27T15:47:39.3635016Z 2022-09-27T15:47:39.3635553Z ---------------------------------------------------------------------- 2022-09-27T15:47:39.3635820Z Ran 1 test in 2.460s 2022-09-27T15:47:39.3635925Z 2022-09-27T15:47:39.3635986Z OK 2022-09-27T15:47:39.3636082Z 2022-09-27T15:47:39.3636177Z Generating XML reports... 2022-09-27T15:47:39.3670872Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154736.xml 2022-09-27T15:47:40.7289986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:40.7290380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:40.7298143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:40.7298499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:40.8865525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaq_3b921 2022-09-27T15:47:40.8866420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaq_3b921/_remote_module_non_scriptable.py 2022-09-27T15:47:41.1451850Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:41.1461524Z 2022-09-27T15:47:41.1461623Z Running tests... 2022-09-27T15:47:41.1462087Z ---------------------------------------------------------------------- 2022-09-27T15:47:41.3923774Z test_nonzero (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21359 2022-09-27T15:47:41.3950427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21360 2022-09-27T15:47:41.3978333Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21361 2022-09-27T15:47:41.4015151Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21362 2022-09-27T15:47:42.5123032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:42.5123652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:42.5378617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:42.5379183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:42.5477800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:42.5478482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:42.5738645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:42.5739336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:42.5791792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:42.5792423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:42.5869330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:42.5869990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:42.6172034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:42.6239201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:42.6240503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:42.6241043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:42.6752321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8owei_l 2022-09-27T15:47:42.6753108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8owei_l/_remote_module_non_scriptable.py 2022-09-27T15:47:42.7011899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hkiwmsn 2022-09-27T15:47:42.7013367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hkiwmsn/_remote_module_non_scriptable.py 2022-09-27T15:47:42.7454479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpji2wbdld 2022-09-27T15:47:42.7455382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpji2wbdld/_remote_module_non_scriptable.py 2022-09-27T15:47:42.7499876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zg6gjx1 2022-09-27T15:47:42.7501110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zg6gjx1/_remote_module_non_scriptable.py 2022-09-27T15:47:42.9329959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:47:42.9565033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:43.0014484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:47:43.0052161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:47:43.6070257Z ok (2.460s) 2022-09-27T15:47:43.6070527Z 2022-09-27T15:47:43.6071045Z ---------------------------------------------------------------------- 2022-09-27T15:47:43.6071362Z Ran 1 test in 2.461s 2022-09-27T15:47:43.6071489Z 2022-09-27T15:47:43.6071552Z OK 2022-09-27T15:47:43.6071644Z 2022-09-27T15:47:43.6071742Z Generating XML reports... 2022-09-27T15:47:43.6108721Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154741.xml 2022-09-27T15:47:44.9691648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:44.9692086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:44.9699761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:44.9700180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:45.1252496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf4k_hfu8 2022-09-27T15:47:45.1253469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf4k_hfu8/_remote_module_non_scriptable.py 2022-09-27T15:47:45.3883218Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:45.3892613Z 2022-09-27T15:47:45.3892739Z Running tests... 2022-09-27T15:47:45.3893317Z ---------------------------------------------------------------------- 2022-09-27T15:47:45.6349922Z test_owner_equality (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21582 2022-09-27T15:47:45.6376642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21583 2022-09-27T15:47:45.6405252Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21584 2022-09-27T15:47:45.6441282Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21585 2022-09-27T15:47:46.7111112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:46.7111843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:46.7431581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:46.7432303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:46.7465504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:46.7466196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:46.7638981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:46.7639668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:46.7640452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:46.7641088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:46.7801553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:46.7802134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:46.8011530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:46.8012246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:46.8013066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:46.8013687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:46.8731812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoa0ovfzd 2022-09-27T15:47:46.8732584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoa0ovfzd/_remote_module_non_scriptable.py 2022-09-27T15:47:46.9083253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptd9qr13 2022-09-27T15:47:46.9084879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptd9qr13/_remote_module_non_scriptable.py 2022-09-27T15:47:46.9291205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynn4cjij 2022-09-27T15:47:46.9291838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpykaz352w 2022-09-27T15:47:46.9292593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynn4cjij/_remote_module_non_scriptable.py 2022-09-27T15:47:46.9293246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpykaz352w/_remote_module_non_scriptable.py 2022-09-27T15:47:47.1306854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:47:47.1654730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:47:47.1847480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:47:47.1849678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:47.8499258Z ok (2.460s) 2022-09-27T15:47:47.8499502Z 2022-09-27T15:47:47.8500014Z ---------------------------------------------------------------------- 2022-09-27T15:47:47.8500456Z Ran 1 test in 2.461s 2022-09-27T15:47:47.8500577Z 2022-09-27T15:47:47.8500624Z OK 2022-09-27T15:47:47.8500716Z 2022-09-27T15:47:47.8500810Z Generating XML reports... 2022-09-27T15:47:47.8536144Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154745.xml 2022-09-27T15:47:49.2346070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:49.2346450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:49.2354767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:49.2355333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:49.3930829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqf3xxvw 2022-09-27T15:47:49.3931743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqf3xxvw/_remote_module_non_scriptable.py 2022-09-27T15:47:49.6529100Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:49.6539016Z 2022-09-27T15:47:49.6539130Z Running tests... 2022-09-27T15:47:49.6539691Z ---------------------------------------------------------------------- 2022-09-27T15:47:49.8985317Z test_owner_rref_backward (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21805 2022-09-27T15:47:49.9012401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21806 2022-09-27T15:47:49.9040676Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21807 2022-09-27T15:47:49.9069962Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21808 2022-09-27T15:47:50.9957140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:50.9957825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:51.0309705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:51.0310380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:51.0313590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:51.0314156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:51.0496104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:51.0496773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:51.0685413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:51.0686117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:51.0774594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:51.0775168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:51.0878048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:51.0878682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:51.1144045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:51.1144735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:51.1584267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg7yobte 2022-09-27T15:47:51.1585190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg7yobte/_remote_module_non_scriptable.py 2022-09-27T15:47:51.1949935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvoc39_jv 2022-09-27T15:47:51.1950802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvoc39_jv/_remote_module_non_scriptable.py 2022-09-27T15:47:51.2139446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp943zwok3 2022-09-27T15:47:51.2140542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp943zwok3/_remote_module_non_scriptable.py 2022-09-27T15:47:51.2417217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jud1usg 2022-09-27T15:47:51.2418350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jud1usg/_remote_module_non_scriptable.py 2022-09-27T15:47:51.4142899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:47:51.4516032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:47:51.4685192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:47:51.4988783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:52.2126839Z ok (2.558s) 2022-09-27T15:47:52.2127074Z 2022-09-27T15:47:52.2127536Z ---------------------------------------------------------------------- 2022-09-27T15:47:52.2128004Z Ran 1 test in 2.559s 2022-09-27T15:47:52.2128170Z 2022-09-27T15:47:52.2128233Z OK 2022-09-27T15:47:52.2128330Z 2022-09-27T15:47:52.2128426Z Generating XML reports... 2022-09-27T15:47:52.2163464Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154749.xml 2022-09-27T15:47:53.5678821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:53.5679207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:53.5686619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:53.5687138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:53.7273981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5pmx1ald 2022-09-27T15:47:53.7274967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5pmx1ald/_remote_module_non_scriptable.py 2022-09-27T15:47:53.9888844Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:53.9898771Z 2022-09-27T15:47:53.9898892Z Running tests... 2022-09-27T15:47:53.9899352Z ---------------------------------------------------------------------- 2022-09-27T15:47:54.2349721Z test_pass_local_rrefs (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22048 2022-09-27T15:47:54.2376531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22049 2022-09-27T15:47:54.2404759Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22050 2022-09-27T15:47:54.2435537Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22051 2022-09-27T15:47:55.3281493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:55.3282166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:55.3564892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:55.3565538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:55.3636774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:55.3637428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:55.3702900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:55.3703557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:55.3802833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:55.3803491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:55.3933890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:55.3934407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:55.4080398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:55.4081337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:55.4170330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:55.4171207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:55.4898338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhsjd9b2 2022-09-27T15:47:55.4899151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhsjd9b2/_remote_module_non_scriptable.py 2022-09-27T15:47:55.5196454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9n18pp2 2022-09-27T15:47:55.5197307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9n18pp2/_remote_module_non_scriptable.py 2022-09-27T15:47:55.5358244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjra5p4a 2022-09-27T15:47:55.5359487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjra5p4a/_remote_module_non_scriptable.py 2022-09-27T15:47:55.5437949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehkjl9hs 2022-09-27T15:47:55.5439529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehkjl9hs/_remote_module_non_scriptable.py 2022-09-27T15:47:55.7466250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:47:55.7729296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:47:55.7936085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:47:55.7968515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:47:56.4491684Z ok (2.459s) 2022-09-27T15:47:56.4491949Z 2022-09-27T15:47:56.4492455Z ---------------------------------------------------------------------- 2022-09-27T15:47:56.4492706Z Ran 1 test in 2.459s 2022-09-27T15:47:56.4492829Z 2022-09-27T15:47:56.4492901Z OK 2022-09-27T15:47:56.4492994Z 2022-09-27T15:47:56.4494815Z Generating XML reports... 2022-09-27T15:47:56.4528794Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154753.xml 2022-09-27T15:47:57.8417845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:57.8418494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:57.8427107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:57.8427756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:58.0110328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_41k2u2_ 2022-09-27T15:47:58.0110823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_41k2u2_/_remote_module_non_scriptable.py 2022-09-27T15:47:58.2755191Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:47:58.2765855Z 2022-09-27T15:47:58.2766110Z Running tests... 2022-09-27T15:47:58.2766664Z ---------------------------------------------------------------------- 2022-09-27T15:47:58.5224027Z test_pg_init_no_rpc_init (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22271 2022-09-27T15:47:58.5252231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22272 2022-09-27T15:47:58.5280202Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22273 2022-09-27T15:47:58.5309477Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22274 2022-09-27T15:47:59.6320385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:59.6321336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:59.6688334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:59.6689325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:59.6762643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:59.6763312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:59.6784662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:59.6785299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:59.6947772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:47:59.6948298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:47:59.7157897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:59.7158645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:59.7180751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:59.7181394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:59.7339092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:47:59.7339837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:47:59.7959300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9a8vdpvk 2022-09-27T15:47:59.7960120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9a8vdpvk/_remote_module_non_scriptable.py 2022-09-27T15:47:59.8419811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hcy2h26 2022-09-27T15:47:59.8420606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hcy2h26/_remote_module_non_scriptable.py 2022-09-27T15:47:59.8442369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp551rwhv0 2022-09-27T15:47:59.8443762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp551rwhv0/_remote_module_non_scriptable.py 2022-09-27T15:47:59.8598743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj9fmxwmr 2022-09-27T15:47:59.8600290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj9fmxwmr/_remote_module_non_scriptable.py 2022-09-27T15:48:00.0705374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:48:00.1008476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:48:00.1009151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:48:00.1220532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:48:00.1372766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:48:00.1473098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:48:00.1474844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:48:00.1475568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:48:00.1476450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:48:00.1478534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:48:00.1479578Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:48:00.1480501Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:48:00.5362821Z ok (2.259s) 2022-09-27T15:48:00.5363055Z 2022-09-27T15:48:00.5363710Z ---------------------------------------------------------------------- 2022-09-27T15:48:00.5364097Z Ran 1 test in 2.260s 2022-09-27T15:48:00.5364214Z 2022-09-27T15:48:00.5364262Z OK 2022-09-27T15:48:00.5364356Z 2022-09-27T15:48:00.5364454Z Generating XML reports... 2022-09-27T15:48:00.5400577Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154758.xml 2022-09-27T15:48:01.9123408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:01.9123795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:01.9133357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:01.9133865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:02.0782006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf38nq588 2022-09-27T15:48:02.0782658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf38nq588/_remote_module_non_scriptable.py 2022-09-27T15:48:02.3426981Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:48:02.3436574Z 2022-09-27T15:48:02.3436940Z Running tests... 2022-09-27T15:48:02.3437335Z ---------------------------------------------------------------------- 2022-09-27T15:48:02.5907035Z test_pickle_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22342 2022-09-27T15:48:02.5935329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22343 2022-09-27T15:48:02.5963980Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22344 2022-09-27T15:48:02.5995439Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22345 2022-09-27T15:48:03.7113836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:03.7114463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:03.7185280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:03.7185949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:03.7483382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:03.7483954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:03.7497049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:03.7497697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:03.7503430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:03.7504053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:03.7553810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:03.7554474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:03.7862247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:03.7862784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:03.7865701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:03.7866388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:03.8757889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxu5h55f 2022-09-27T15:48:03.8759352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxu5h55f/_remote_module_non_scriptable.py 2022-09-27T15:48:03.8808301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5d_tw2a 2022-09-27T15:48:03.8809707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5d_tw2a/_remote_module_non_scriptable.py 2022-09-27T15:48:03.9122356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjd3bh4a_ 2022-09-27T15:48:03.9123341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjd3bh4a_/_remote_module_non_scriptable.py 2022-09-27T15:48:03.9125289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4j9q0uw 2022-09-27T15:48:03.9127883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4j9q0uw/_remote_module_non_scriptable.py 2022-09-27T15:48:04.1373486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:48:04.1405354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:48:04.1710430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:48:04.1897657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:48:04.9053526Z ok (2.561s) 2022-09-27T15:48:04.9053761Z 2022-09-27T15:48:04.9054276Z ---------------------------------------------------------------------- 2022-09-27T15:48:04.9054694Z Ran 1 test in 2.562s 2022-09-27T15:48:04.9054816Z 2022-09-27T15:48:04.9054867Z OK 2022-09-27T15:48:04.9054959Z 2022-09-27T15:48:04.9055057Z Generating XML reports... 2022-09-27T15:48:04.9092286Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154802.xml 2022-09-27T15:48:06.2770848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:06.2771245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:06.2778760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:06.2779232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:06.4328109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpywh0tddj 2022-09-27T15:48:06.4328852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpywh0tddj/_remote_module_non_scriptable.py 2022-09-27T15:48:06.6960636Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:48:06.6970172Z 2022-09-27T15:48:06.6970334Z Running tests... 2022-09-27T15:48:06.6971038Z ---------------------------------------------------------------------- 2022-09-27T15:48:06.9395758Z test_profiler_export_trace (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22565 2022-09-27T15:48:06.9423324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22566 2022-09-27T15:48:06.9451265Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22567 2022-09-27T15:48:06.9481908Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22568 2022-09-27T15:48:08.1034727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:08.1035385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:08.1047967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:08.1048859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:08.1078529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:08.1079407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:08.1097890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:08.1098538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:08.1416668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:08.1417312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:08.1432965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:08.1433724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:08.1463620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:08.1464366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:08.1476494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:08.1477044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:08.2693280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp191ss0fb 2022-09-27T15:48:08.2694380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp191ss0fb/_remote_module_non_scriptable.py 2022-09-27T15:48:08.2726271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxi1lc2hd 2022-09-27T15:48:08.2727434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxi1lc2hd/_remote_module_non_scriptable.py 2022-09-27T15:48:08.2741254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuejpv72t 2022-09-27T15:48:08.2743275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuejpv72t/_remote_module_non_scriptable.py 2022-09-27T15:48:08.2906828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8enypk7 2022-09-27T15:48:08.2908643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8enypk7/_remote_module_non_scriptable.py 2022-09-27T15:48:08.5255690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:48:08.5279738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:48:08.5313101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:48:08.5464701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:48:09.1537202Z ok (2.456s) 2022-09-27T15:48:09.1537441Z 2022-09-27T15:48:09.1538129Z ---------------------------------------------------------------------- 2022-09-27T15:48:09.1538469Z Ran 1 test in 2.457s 2022-09-27T15:48:09.1538621Z 2022-09-27T15:48:09.1538677Z OK 2022-09-27T15:48:09.1538771Z 2022-09-27T15:48:09.1538866Z Generating XML reports... 2022-09-27T15:48:09.1573422Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154806.xml 2022-09-27T15:48:10.5333096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:10.5333494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:10.5341254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:10.5341596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:10.6895524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7ynl1lf 2022-09-27T15:48:10.6896909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7ynl1lf/_remote_module_non_scriptable.py 2022-09-27T15:48:10.9515867Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:48:10.9525093Z 2022-09-27T15:48:10.9525534Z Running tests... 2022-09-27T15:48:10.9525966Z ---------------------------------------------------------------------- 2022-09-27T15:48:11.1945283Z test_profiler_remote_events_profiled (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22788 2022-09-27T15:48:11.1973243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22789 2022-09-27T15:48:11.2000792Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22790 2022-09-27T15:48:11.2030664Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22791 2022-09-27T15:48:12.3251523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:12.3252265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:12.3412327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:12.3413000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:12.3481202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:12.3481812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:12.3656779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:12.3657285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:12.3799636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:12.3800242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:12.3854301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:12.3854931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:12.3855732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:12.3856295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:12.4203762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:12.4204416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:12.4925720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyp0c5804 2022-09-27T15:48:12.4926532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyp0c5804/_remote_module_non_scriptable.py 2022-09-27T15:48:12.5071170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptkams9lb 2022-09-27T15:48:12.5072461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptkams9lb/_remote_module_non_scriptable.py 2022-09-27T15:48:12.5120978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xzghwx3 2022-09-27T15:48:12.5122817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xzghwx3/_remote_module_non_scriptable.py 2022-09-27T15:48:12.5452600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgq5wrqzz 2022-09-27T15:48:12.5453649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgq5wrqzz/_remote_module_non_scriptable.py 2022-09-27T15:48:12.7491870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:48:12.7647230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:48:12.7696151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:48:12.8027921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:48:13.4086748Z ok (2.456s) 2022-09-27T15:48:13.4086948Z 2022-09-27T15:48:13.4087409Z ---------------------------------------------------------------------- 2022-09-27T15:48:13.4087871Z Ran 1 test in 2.456s 2022-09-27T15:48:13.4088057Z 2022-09-27T15:48:13.4088105Z OK 2022-09-27T15:48:13.4088202Z 2022-09-27T15:48:13.4088298Z Generating XML reports... 2022-09-27T15:48:13.4124784Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154810.xml 2022-09-27T15:48:14.8178542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:14.8179175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:14.8187474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:14.8188099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:14.9743906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51ikan2n 2022-09-27T15:48:14.9744593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51ikan2n/_remote_module_non_scriptable.py 2022-09-27T15:48:15.2346493Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:48:15.2356770Z 2022-09-27T15:48:15.2357340Z Running tests... 2022-09-27T15:48:15.2357900Z ---------------------------------------------------------------------- 2022-09-27T15:48:15.4791620Z test_profiler_remote_events_profiled_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23011 2022-09-27T15:48:15.4817248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23012 2022-09-27T15:48:15.4845135Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23013 2022-09-27T15:48:15.4875267Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23014 2022-09-27T15:48:16.5710845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:16.5711525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:16.6056259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:16.6056808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:16.6062096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:16.6062742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:16.6138860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:16.6139372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:16.6165607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:16.6166115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:16.6456095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:16.6456696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:16.6533603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:16.6534638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:16.6553138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:16.6553944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:16.7328486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppoxowq01 2022-09-27T15:48:16.7329281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppoxowq01/_remote_module_non_scriptable.py 2022-09-27T15:48:16.7729118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1gbks03 2022-09-27T15:48:16.7729888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1gbks03/_remote_module_non_scriptable.py 2022-09-27T15:48:16.7797953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaz5t2c2x 2022-09-27T15:48:16.7799452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaz5t2c2x/_remote_module_non_scriptable.py 2022-09-27T15:48:16.7815784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5ocsvcg 2022-09-27T15:48:16.7817355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5ocsvcg/_remote_module_non_scriptable.py 2022-09-27T15:48:16.9897754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:48:17.0311576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:48:17.0380702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:48:17.0381329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:48:17.6931456Z ok (2.457s) 2022-09-27T15:48:17.6931709Z 2022-09-27T15:48:17.6932236Z ---------------------------------------------------------------------- 2022-09-27T15:48:17.6932535Z Ran 1 test in 2.457s 2022-09-27T15:48:17.6932653Z 2022-09-27T15:48:17.6932725Z OK 2022-09-27T15:48:17.6932820Z 2022-09-27T15:48:17.6932913Z Generating XML reports... 2022-09-27T15:48:17.6968666Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154815.xml 2022-09-27T15:48:19.0413121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:19.0413671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:19.0436311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:19.0436701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:19.2006371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpee8tpp3s 2022-09-27T15:48:19.2007080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpee8tpp3s/_remote_module_non_scriptable.py 2022-09-27T15:48:19.4630026Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:48:19.4640091Z 2022-09-27T15:48:19.4640304Z Running tests... 2022-09-27T15:48:19.4640662Z ---------------------------------------------------------------------- 2022-09-27T15:48:19.7040162Z test_profiler_rpc_key_names (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23234 2022-09-27T15:48:19.7066277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23235 2022-09-27T15:48:19.7094032Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23236 2022-09-27T15:48:19.7123206Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23237 2022-09-27T15:48:20.8493084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:20.8494047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:20.8563968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:20.8564918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:20.8640408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:20.8641047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:20.8648281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:20.8648903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:20.8886121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:20.8886695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:20.8969519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:20.8970160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:20.9037300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:20.9038100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:20.9052064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:20.9053008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:21.0155672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchrj64wb 2022-09-27T15:48:21.0156380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchrj64wb/_remote_module_non_scriptable.py 2022-09-27T15:48:21.0234753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjs7nr5nc 2022-09-27T15:48:21.0237153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjs7nr5nc/_remote_module_non_scriptable.py 2022-09-27T15:48:21.0304745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpjb3rc3 2022-09-27T15:48:21.0306927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpjb3rc3/_remote_module_non_scriptable.py 2022-09-27T15:48:21.0320266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvgy_fhu 2022-09-27T15:48:21.0322172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvgy_fhu/_remote_module_non_scriptable.py 2022-09-27T15:48:21.2717240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:48:21.2777612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:48:21.2854068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:48:21.2909266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:48:21.9178866Z ok (2.453s) 2022-09-27T15:48:21.9179221Z 2022-09-27T15:48:21.9179758Z ---------------------------------------------------------------------- 2022-09-27T15:48:21.9180081Z Ran 1 test in 2.454s 2022-09-27T15:48:21.9180197Z 2022-09-27T15:48:21.9180257Z OK 2022-09-27T15:48:21.9180348Z 2022-09-27T15:48:21.9180440Z Generating XML reports... 2022-09-27T15:48:21.9217078Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154819.xml 2022-09-27T15:48:23.2966258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:23.2966637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:23.2974515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:23.2974883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:23.4538583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphpuqclsq 2022-09-27T15:48:23.4539714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphpuqclsq/_remote_module_non_scriptable.py 2022-09-27T15:48:23.7161269Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:48:23.7170911Z 2022-09-27T15:48:23.7171334Z Running tests... 2022-09-27T15:48:23.7171777Z ---------------------------------------------------------------------- 2022-09-27T15:48:23.9596907Z test_profiler_rpc_memory (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23463 2022-09-27T15:48:23.9625834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23464 2022-09-27T15:48:23.9654642Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23465 2022-09-27T15:48:23.9685660Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23466 2022-09-27T15:48:25.0942717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:25.0943361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:25.1176534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:25.1177137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:25.1290482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:25.1291121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:25.1533899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:25.1534546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:25.1605074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:25.1605729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:25.1954891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:25.1955561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:25.1970499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:25.1971185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:25.2332172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:25.2332799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:25.2559855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcmrc9ne 2022-09-27T15:48:25.2560678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcmrc9ne/_remote_module_non_scriptable.py 2022-09-27T15:48:25.2810859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtk4n5eb 2022-09-27T15:48:25.2811623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtk4n5eb/_remote_module_non_scriptable.py 2022-09-27T15:48:25.3266427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbipak1h 2022-09-27T15:48:25.3267228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbipak1h/_remote_module_non_scriptable.py 2022-09-27T15:48:25.3589526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4nqyy1n 2022-09-27T15:48:25.3590529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4nqyy1n/_remote_module_non_scriptable.py 2022-09-27T15:48:25.5134457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:48:25.5397215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:48:25.5898847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:48:25.6196293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:48:26.2743076Z ok (2.557s) 2022-09-27T15:48:26.2743307Z 2022-09-27T15:48:26.2743710Z ---------------------------------------------------------------------- 2022-09-27T15:48:26.2743949Z Ran 1 test in 2.557s 2022-09-27T15:48:26.2744068Z 2022-09-27T15:48:26.2744130Z OK 2022-09-27T15:48:26.2744223Z 2022-09-27T15:48:26.2744322Z Generating XML reports... 2022-09-27T15:48:26.2779619Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154823.xml 2022-09-27T15:48:27.6859282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:27.6859678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:27.6867888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:27.6868394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:27.8480180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvniexiw6 2022-09-27T15:48:27.8480970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvniexiw6/_remote_module_non_scriptable.py 2022-09-27T15:48:28.1093773Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:48:28.1103364Z 2022-09-27T15:48:28.1103464Z Running tests... 2022-09-27T15:48:28.1104045Z ---------------------------------------------------------------------- 2022-09-27T15:48:28.3527280Z test_profiler_rpc_record_shapes (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23686 2022-09-27T15:48:28.3553519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23687 2022-09-27T15:48:28.3581596Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23688 2022-09-27T15:48:28.3611722Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23689 2022-09-27T15:48:29.4340601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:29.4341252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:29.4595565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:29.4596241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:29.4716297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:29.4716827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:29.4877509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:29.4878159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:29.4968378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:29.4968992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:29.5152634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:29.5153273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:29.5250978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:29.5251664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:29.5527778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:29.6000069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:29.6000738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeoioabd8 2022-09-27T15:48:29.6001464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeoioabd8/_remote_module_non_scriptable.py 2022-09-27T15:48:29.6241533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zxhzei0 2022-09-27T15:48:29.6242368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zxhzei0/_remote_module_non_scriptable.py 2022-09-27T15:48:29.6513542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv44i3ijz 2022-09-27T15:48:29.6514838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv44i3ijz/_remote_module_non_scriptable.py 2022-09-27T15:48:29.6794420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9upcwx1r 2022-09-27T15:48:29.6795568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9upcwx1r/_remote_module_non_scriptable.py 2022-09-27T15:48:29.8565513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:48:29.8794736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:48:29.9091143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:48:29.9373794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:48:30.5667601Z ok (2.456s) 2022-09-27T15:48:30.5667840Z 2022-09-27T15:48:30.5668371Z ---------------------------------------------------------------------- 2022-09-27T15:48:30.5668721Z Ran 1 test in 2.456s 2022-09-27T15:48:30.5668825Z 2022-09-27T15:48:30.5668893Z OK 2022-09-27T15:48:30.5668984Z 2022-09-27T15:48:30.5669078Z Generating XML reports... 2022-09-27T15:48:30.5704631Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154828.xml 2022-09-27T15:48:31.9454591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:31.9454965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:31.9464534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:31.9464901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:32.1091345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64s7_snw 2022-09-27T15:48:32.1092839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64s7_snw/_remote_module_non_scriptable.py 2022-09-27T15:48:32.3705476Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:48:32.3715468Z 2022-09-27T15:48:32.3715558Z Running tests... 2022-09-27T15:48:32.3716508Z ---------------------------------------------------------------------- 2022-09-27T15:48:32.6122911Z test_profiler_with_async_rpc_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23909 2022-09-27T15:48:32.6151033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23910 2022-09-27T15:48:32.6179046Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23911 2022-09-27T15:48:32.6208884Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23912 2022-09-27T15:48:33.6966279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:33.6966933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:33.6976668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:33.6977356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:33.7325666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:33.7326416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:33.7337887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:33.7338528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:33.7454966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:33.7455572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:33.7506016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:33.7506655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:33.7808590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:33.7809180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:33.7869062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:33.7869709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:33.8595340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvdv_ad6c 2022-09-27T15:48:33.8596617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvdv_ad6c/_remote_module_non_scriptable.py 2022-09-27T15:48:33.8600168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_h2t6pxh 2022-09-27T15:48:33.8602298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_h2t6pxh/_remote_module_non_scriptable.py 2022-09-27T15:48:33.9063545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9a95k3vy 2022-09-27T15:48:33.9064753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9a95k3vy/_remote_module_non_scriptable.py 2022-09-27T15:48:33.9132691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2h77xe9 2022-09-27T15:48:33.9133651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2h77xe9/_remote_module_non_scriptable.py 2022-09-27T15:48:34.1156294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:48:34.1166253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:48:34.1637991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:48:34.1875184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:48:34.8263808Z ok (2.454s) 2022-09-27T15:48:34.8264290Z 2022-09-27T15:48:34.8265220Z ---------------------------------------------------------------------- 2022-09-27T15:48:34.8265719Z Ran 1 test in 2.455s 2022-09-27T15:48:34.8265844Z 2022-09-27T15:48:34.8265904Z OK 2022-09-27T15:48:34.8265997Z 2022-09-27T15:48:34.8266095Z Generating XML reports... 2022-09-27T15:48:34.8300782Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154832.xml 2022-09-27T15:48:36.1715729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:36.1716386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:36.1724717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:36.1725309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:36.3287986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vi0nhry 2022-09-27T15:48:36.3289074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vi0nhry/_remote_module_non_scriptable.py 2022-09-27T15:48:36.5887020Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:48:36.5896595Z 2022-09-27T15:48:36.5896721Z Running tests... 2022-09-27T15:48:36.5897189Z ---------------------------------------------------------------------- 2022-09-27T15:48:36.8330902Z test_profiler_with_async_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24132 2022-09-27T15:48:36.8359144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24133 2022-09-27T15:48:36.8386870Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24134 2022-09-27T15:48:36.8417537Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24135 2022-09-27T15:48:38.0008585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:38.0009241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:38.0010005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:38.0010528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:38.0392674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:38.0393272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:38.0393991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:38.0394772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:38.0558795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:38.0559379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:38.0601853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:38.0602489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:38.0946383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:38.0947146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:38.0991774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:38.0992409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:38.1665413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_1oe7zy 2022-09-27T15:48:38.1667829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_1oe7zy/_remote_module_non_scriptable.py 2022-09-27T15:48:38.1861634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpugnj6fds 2022-09-27T15:48:38.1862443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpugnj6fds/_remote_module_non_scriptable.py 2022-09-27T15:48:38.2255555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplfh44g0d 2022-09-27T15:48:38.2256277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplfh44g0d/_remote_module_non_scriptable.py 2022-09-27T15:48:38.2299676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmmt6gn4 2022-09-27T15:48:38.2300817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmmt6gn4/_remote_module_non_scriptable.py 2022-09-27T15:48:38.4257623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:48:38.4460599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:48:38.4820686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:48:38.4901171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:48:39.1475805Z ok (2.557s) 2022-09-27T15:48:39.1476070Z 2022-09-27T15:48:39.1476569Z ---------------------------------------------------------------------- 2022-09-27T15:48:39.1476818Z Ran 1 test in 2.558s 2022-09-27T15:48:39.1476934Z 2022-09-27T15:48:39.1476995Z OK 2022-09-27T15:48:39.1477085Z 2022-09-27T15:48:39.1477195Z Generating XML reports... 2022-09-27T15:48:39.1511845Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154836.xml 2022-09-27T15:48:40.5059379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:40.5059742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:40.5067539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:40.5067900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:40.6612406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb543xp9c 2022-09-27T15:48:40.6613387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb543xp9c/_remote_module_non_scriptable.py 2022-09-27T15:48:40.9207510Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:48:40.9217463Z 2022-09-27T15:48:40.9217552Z Running tests... 2022-09-27T15:48:40.9218533Z ---------------------------------------------------------------------- 2022-09-27T15:48:41.1638027Z test_profiler_with_async_rpc_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24355 2022-09-27T15:48:41.1666275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24356 2022-09-27T15:48:41.1694025Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24357 2022-09-27T15:48:41.1724869Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24358 2022-09-27T15:48:42.2830600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:42.2831340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:42.2839069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:42.2839701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:42.3007268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:42.3007952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:42.3124062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:42.3124710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:42.3222842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:42.3223475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:42.3236216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:42.3236847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:42.3394143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:42.3395000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:42.3497219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:42.3497870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:42.4495457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzzezgcnv 2022-09-27T15:48:42.4496231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkudh6yh 2022-09-27T15:48:42.4496960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzzezgcnv/_remote_module_non_scriptable.py 2022-09-27T15:48:42.4497740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkudh6yh/_remote_module_non_scriptable.py 2022-09-27T15:48:42.4664106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgk7int4k 2022-09-27T15:48:42.4665426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgk7int4k/_remote_module_non_scriptable.py 2022-09-27T15:48:42.4756322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90krgc14 2022-09-27T15:48:42.4757438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90krgc14/_remote_module_non_scriptable.py 2022-09-27T15:48:42.7061545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:48:42.7070630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:48:42.7243626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:48:42.7328789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:48:44.9591579Z STAGE:2022-09-27 15:48:44 24356:24356 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-09-27T15:48:44.9592443Z [W utils.cpp:175] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator()) 2022-09-27T15:48:45.9620468Z STAGE:2022-09-27 15:48:45 24356:24356 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-09-27T15:48:46.3829256Z ok (5.461s) 2022-09-27T15:48:46.3829477Z 2022-09-27T15:48:46.3829975Z ---------------------------------------------------------------------- 2022-09-27T15:48:46.3830346Z Ran 1 test in 5.461s 2022-09-27T15:48:46.3830464Z 2022-09-27T15:48:46.3830524Z OK 2022-09-27T15:48:46.3830614Z 2022-09-27T15:48:46.3830706Z Generating XML reports... 2022-09-27T15:48:46.3866206Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154840.xml 2022-09-27T15:48:47.7547138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:47.7547523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:47.7556209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:47.7556764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:47.9162330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbrwmuo9t 2022-09-27T15:48:47.9162832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbrwmuo9t/_remote_module_non_scriptable.py 2022-09-27T15:48:48.1789165Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:48:48.1798600Z 2022-09-27T15:48:48.1798714Z Running tests... 2022-09-27T15:48:48.1799381Z ---------------------------------------------------------------------- 2022-09-27T15:48:48.4265736Z test_profiler_with_async_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24579 2022-09-27T15:48:48.4293676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24580 2022-09-27T15:48:48.4322034Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24581 2022-09-27T15:48:48.4353517Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24582 2022-09-27T15:48:49.5070723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:49.5071471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:49.5205942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:49.5206584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:49.5350685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:49.5351453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:49.5459466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:49.5460035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:49.5534324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:49.5534859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:49.5595302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:49.5747172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:49.5748100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:49.5748633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:49.5923048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:49.5923697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:49.6792931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg39zrnf5 2022-09-27T15:48:49.6793776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg39zrnf5/_remote_module_non_scriptable.py 2022-09-27T15:48:49.7062395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5tuqqz7 2022-09-27T15:48:49.7063184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6v8vi37 2022-09-27T15:48:49.7063898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5tuqqz7/_remote_module_non_scriptable.py 2022-09-27T15:48:49.7064624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6v8vi37/_remote_module_non_scriptable.py 2022-09-27T15:48:49.7228107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsry1_dsu 2022-09-27T15:48:49.7229107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsry1_dsu/_remote_module_non_scriptable.py 2022-09-27T15:48:49.9356810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:48:49.9609712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:48:49.9627936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:48:49.9777742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:48:52.2531056Z STAGE:2022-09-27 15:48:52 24580:24580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-09-27T15:48:52.2532407Z [W utils.cpp:175] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator()) 2022-09-27T15:48:53.2564801Z STAGE:2022-09-27 15:48:53 24580:24580 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-09-27T15:48:53.6456643Z ok (5.465s) 2022-09-27T15:48:53.6456789Z 2022-09-27T15:48:53.6457203Z ---------------------------------------------------------------------- 2022-09-27T15:48:53.6457476Z Ran 1 test in 5.466s 2022-09-27T15:48:53.6457596Z 2022-09-27T15:48:53.6457659Z OK 2022-09-27T15:48:53.6457755Z 2022-09-27T15:48:53.6457850Z Generating XML reports... 2022-09-27T15:48:53.6493382Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154848.xml 2022-09-27T15:48:54.9915755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:54.9916149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:54.9924551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:54.9924910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:55.1450905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4q_r2tca 2022-09-27T15:48:55.1451830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4q_r2tca/_remote_module_non_scriptable.py 2022-09-27T15:48:55.4048110Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:48:55.4057773Z 2022-09-27T15:48:55.4057875Z Running tests... 2022-09-27T15:48:55.4058327Z ---------------------------------------------------------------------- 2022-09-27T15:48:55.6458383Z test_profiler_with_autograd_context (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24803 2022-09-27T15:48:55.6487465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24804 2022-09-27T15:48:55.6515945Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24805 2022-09-27T15:48:55.6545773Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24806 2022-09-27T15:48:56.7777835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:56.7778452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:56.7986336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:56.7986952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:56.8129748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:56.8130447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:56.8206036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:56.8206752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:56.8286408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:56.8286944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:56.8364500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:56.8365056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:56.8579661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:56.8580307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:56.8655321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:56.8656047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:56.9391559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2t3dszfw 2022-09-27T15:48:56.9392381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2t3dszfw/_remote_module_non_scriptable.py 2022-09-27T15:48:56.9635847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplsz9r6tw 2022-09-27T15:48:56.9636695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplsz9r6tw/_remote_module_non_scriptable.py 2022-09-27T15:48:56.9874229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprthvts8u 2022-09-27T15:48:56.9875166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprthvts8u/_remote_module_non_scriptable.py 2022-09-27T15:48:57.0105355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazbss88q 2022-09-27T15:48:57.0106178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazbss88q/_remote_module_non_scriptable.py 2022-09-27T15:48:57.1959925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:48:57.2216258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:48:57.2423391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:48:57.2637042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:48:57.8601475Z ok (2.454s) 2022-09-27T15:48:57.8601768Z 2022-09-27T15:48:57.8602310Z ---------------------------------------------------------------------- 2022-09-27T15:48:57.8602621Z Ran 1 test in 2.454s 2022-09-27T15:48:57.8602740Z 2022-09-27T15:48:57.8602802Z OK 2022-09-27T15:48:57.8602882Z 2022-09-27T15:48:57.8602993Z Generating XML reports... 2022-09-27T15:48:57.8637258Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154855.xml 2022-09-27T15:48:59.2332997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:48:59.2333373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:48:59.2341068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:48:59.2341465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:48:59.3881004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1c7xjyg 2022-09-27T15:48:59.3882119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1c7xjyg/_remote_module_non_scriptable.py 2022-09-27T15:48:59.6476065Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:48:59.6485894Z 2022-09-27T15:48:59.6486274Z Running tests... 2022-09-27T15:48:59.6486735Z ---------------------------------------------------------------------- 2022-09-27T15:48:59.8907626Z test_profiler_with_autograd_context_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25026 2022-09-27T15:48:59.8934969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25027 2022-09-27T15:48:59.8962418Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25028 2022-09-27T15:48:59.8991890Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25029 2022-09-27T15:49:01.0602447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:01.0603078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:01.0824859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:01.0825559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:01.0886923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:01.0887531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:01.0970126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:01.0970788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:01.0991374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:01.0992034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:01.1222847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:01.1223419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:01.1355963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:01.1356578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:01.1393590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:01.1394240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:01.2303498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzojmaud 2022-09-27T15:49:01.2304457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzojmaud/_remote_module_non_scriptable.py 2022-09-27T15:49:01.2488135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3p7quce1 2022-09-27T15:49:01.2488974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3p7quce1/_remote_module_non_scriptable.py 2022-09-27T15:49:01.2621725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu920y404 2022-09-27T15:49:01.2622770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu920y404/_remote_module_non_scriptable.py 2022-09-27T15:49:01.2654406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp226nsean 2022-09-27T15:49:01.2656432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp226nsean/_remote_module_non_scriptable.py 2022-09-27T15:49:01.4894399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:49:01.5056588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:49:01.5203683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:49:01.5243087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:49:02.2050157Z ok (2.556s) 2022-09-27T15:49:02.2050379Z 2022-09-27T15:49:02.2050710Z ---------------------------------------------------------------------- 2022-09-27T15:49:02.2050953Z Ran 1 test in 2.556s 2022-09-27T15:49:02.2051075Z 2022-09-27T15:49:02.2051139Z OK 2022-09-27T15:49:02.2051233Z 2022-09-27T15:49:02.2051328Z Generating XML reports... 2022-09-27T15:49:02.2088606Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154859.xml 2022-09-27T15:49:03.5965153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:03.5965648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:03.5973862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:03.5974752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:03.7546063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ol0_yy3 2022-09-27T15:49:03.7547098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ol0_yy3/_remote_module_non_scriptable.py 2022-09-27T15:49:04.0138371Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:49:04.0147864Z 2022-09-27T15:49:04.0148124Z Running tests... 2022-09-27T15:49:04.0148740Z ---------------------------------------------------------------------- 2022-09-27T15:49:04.2614229Z test_profiler_with_remote_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25249 2022-09-27T15:49:04.2641301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25250 2022-09-27T15:49:04.2668955Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25251 2022-09-27T15:49:04.2699315Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25252 2022-09-27T15:49:05.3982468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:05.3983137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:05.4332166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:05.4332812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:05.4393607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:05.4542242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:05.4543153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:05.4543674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:05.4645635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:05.4646301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:05.4761167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:05.4761739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:05.4907052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:05.4907711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:05.5006295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:05.5006960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:05.5591748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpui834ehx 2022-09-27T15:49:05.5592590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpui834ehx/_remote_module_non_scriptable.py 2022-09-27T15:49:05.6024865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuubkbhs1 2022-09-27T15:49:05.6025727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuubkbhs1/_remote_module_non_scriptable.py 2022-09-27T15:49:05.6165667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbrnwncl 2022-09-27T15:49:05.6167083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbrnwncl/_remote_module_non_scriptable.py 2022-09-27T15:49:05.6253798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbykyq0hq 2022-09-27T15:49:05.6255337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbykyq0hq/_remote_module_non_scriptable.py 2022-09-27T15:49:05.8138189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:49:05.8639302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:49:05.8762615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:49:05.8815701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:49:06.4755795Z ok (2.460s) 2022-09-27T15:49:06.4755962Z 2022-09-27T15:49:06.4756403Z ---------------------------------------------------------------------- 2022-09-27T15:49:06.4756866Z Ran 1 test in 2.461s 2022-09-27T15:49:06.4757080Z 2022-09-27T15:49:06.4757185Z OK 2022-09-27T15:49:06.4757279Z 2022-09-27T15:49:06.4757379Z Generating XML reports... 2022-09-27T15:49:06.4792382Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154904.xml 2022-09-27T15:49:07.8788608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:07.8788981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:07.8796824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:07.8797188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:08.0365513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67e2ydwj 2022-09-27T15:49:08.0366508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67e2ydwj/_remote_module_non_scriptable.py 2022-09-27T15:49:08.2949371Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:49:08.2959289Z 2022-09-27T15:49:08.2959417Z Running tests... 2022-09-27T15:49:08.2959867Z ---------------------------------------------------------------------- 2022-09-27T15:49:08.5419957Z test_profiler_with_remote_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25472 2022-09-27T15:49:08.5447762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25473 2022-09-27T15:49:08.5475839Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25474 2022-09-27T15:49:08.5506582Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25475 2022-09-27T15:49:09.6679079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:09.6679703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:09.6684732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:09.6685313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:09.6903004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:09.6903707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:09.7066669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:09.7067245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:09.7068265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:09.7068899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:09.7082583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:09.7083172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:09.7280204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:09.7280831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:09.7436252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:09.7436887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:09.8327963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwujwwz8x 2022-09-27T15:49:09.8328720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmaq1fxr 2022-09-27T15:49:09.8329411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwujwwz8x/_remote_module_non_scriptable.py 2022-09-27T15:49:09.8330081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmaq1fxr/_remote_module_non_scriptable.py 2022-09-27T15:49:09.8545838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4_6xd0l 2022-09-27T15:49:09.8547171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4_6xd0l/_remote_module_non_scriptable.py 2022-09-27T15:49:09.8695695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3w17z62v 2022-09-27T15:49:09.8697190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3w17z62v/_remote_module_non_scriptable.py 2022-09-27T15:49:10.0857867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:49:10.0864230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:49:10.1088760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:49:10.1248538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:49:10.8563162Z ok (2.560s) 2022-09-27T15:49:10.8563392Z 2022-09-27T15:49:10.8563717Z ---------------------------------------------------------------------- 2022-09-27T15:49:10.8564007Z Ran 1 test in 2.560s 2022-09-27T15:49:10.8564171Z 2022-09-27T15:49:10.8564255Z OK 2022-09-27T15:49:10.8564351Z 2022-09-27T15:49:10.8564450Z Generating XML reports... 2022-09-27T15:49:10.8600049Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154908.xml 2022-09-27T15:49:12.2227227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:12.2227609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:12.2236244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:12.2236606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:12.3824024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_jfne83 2022-09-27T15:49:12.3825013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_jfne83/_remote_module_non_scriptable.py 2022-09-27T15:49:12.6432496Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:49:12.6442855Z 2022-09-27T15:49:12.6443182Z Running tests... 2022-09-27T15:49:12.6443772Z ---------------------------------------------------------------------- 2022-09-27T15:49:12.8858930Z test_profiler_with_remote_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25695 2022-09-27T15:49:12.8886864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25696 2022-09-27T15:49:12.8914337Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25697 2022-09-27T15:49:12.8944037Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25698 2022-09-27T15:49:14.0237323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:14.0238018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:14.0497412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:14.0497947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:14.0634823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:14.0635457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:14.0881148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:14.0881792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:14.0955579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:14.0956189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:14.1080906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:14.1081532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:14.1334742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:14.1335409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:14.1469125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:14.1469799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:14.1928559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfj9ys7c 2022-09-27T15:49:14.1929386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfj9ys7c/_remote_module_non_scriptable.py 2022-09-27T15:49:14.2164729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjqtzutb 2022-09-27T15:49:14.2165545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjqtzutb/_remote_module_non_scriptable.py 2022-09-27T15:49:14.2606044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkd_18mw 2022-09-27T15:49:14.2606777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkd_18mw/_remote_module_non_scriptable.py 2022-09-27T15:49:14.2750063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8iifdkv8 2022-09-27T15:49:14.2750886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8iifdkv8/_remote_module_non_scriptable.py 2022-09-27T15:49:14.4552805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:49:14.4798946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:49:14.5228006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:49:14.5420026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:49:18.2048155Z ok (5.560s) 2022-09-27T15:49:18.2048372Z 2022-09-27T15:49:18.2048771Z ---------------------------------------------------------------------- 2022-09-27T15:49:18.2049085Z Ran 1 test in 5.560s 2022-09-27T15:49:18.2051073Z 2022-09-27T15:49:18.2051261Z OK 2022-09-27T15:49:18.2051697Z 2022-09-27T15:49:18.2052103Z Generating XML reports... 2022-09-27T15:49:18.2085995Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154912.xml 2022-09-27T15:49:19.5702403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:19.5703029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:19.5711647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:19.5712401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:19.7274868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzu_va2ir 2022-09-27T15:49:19.7275774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzu_va2ir/_remote_module_non_scriptable.py 2022-09-27T15:49:19.9886722Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:49:19.9896843Z 2022-09-27T15:49:19.9897192Z Running tests... 2022-09-27T15:49:19.9897783Z ---------------------------------------------------------------------- 2022-09-27T15:49:20.2279015Z test_profiler_with_remote_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25918 2022-09-27T15:49:20.2306790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25919 2022-09-27T15:49:20.2334998Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25920 2022-09-27T15:49:20.2364741Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25921 2022-09-27T15:49:21.3655289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:21.3655970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:21.3782029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:21.3782723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:21.3896707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:21.3897196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:21.3905453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:21.3906123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:21.4038026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:21.4038563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:21.4165439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:21.4166112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:21.4272192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:21.4272783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:21.4278587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:21.4279207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:21.5304167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4e8pewvw 2022-09-27T15:49:21.5304949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4e8pewvw/_remote_module_non_scriptable.py 2022-09-27T15:49:21.5422114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekn2zc5h 2022-09-27T15:49:21.5423303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekn2zc5h/_remote_module_non_scriptable.py 2022-09-27T15:49:21.5532903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2m78ua36 2022-09-27T15:49:21.5533914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2m78ua36/_remote_module_non_scriptable.py 2022-09-27T15:49:21.5537983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7f3l326v 2022-09-27T15:49:21.5540078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7f3l326v/_remote_module_non_scriptable.py 2022-09-27T15:49:21.7877750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:49:21.7963245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:49:21.8079080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:49:21.8079735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:49:25.5472475Z ok (5.557s) 2022-09-27T15:49:25.5472754Z 2022-09-27T15:49:25.5473257Z ---------------------------------------------------------------------- 2022-09-27T15:49:25.5473704Z Ran 1 test in 5.558s 2022-09-27T15:49:25.5473916Z 2022-09-27T15:49:25.5474026Z OK 2022-09-27T15:49:25.5474178Z 2022-09-27T15:49:25.5474313Z Generating XML reports... 2022-09-27T15:49:25.5510377Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154919.xml 2022-09-27T15:49:26.9140028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:26.9140450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:26.9148637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:26.9149061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:27.0721518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkzk6i41 2022-09-27T15:49:27.0722645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkzk6i41/_remote_module_non_scriptable.py 2022-09-27T15:49:27.3333865Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:49:27.3343488Z 2022-09-27T15:49:27.3343572Z Running tests... 2022-09-27T15:49:27.3344505Z ---------------------------------------------------------------------- 2022-09-27T15:49:27.5774842Z test_profiler_with_script_async_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26141 2022-09-27T15:49:27.5802680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26142 2022-09-27T15:49:27.5828930Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26143 2022-09-27T15:49:27.5858609Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26144 2022-09-27T15:49:28.7206879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:28.7207525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:28.7309028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:28.7309657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:28.7340430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:28.7341099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:28.7601998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:28.7602716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:28.7701311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:28.7702044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:28.7727202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:28.7728072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:28.7800105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:28.7800975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:28.8147571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:28.8148253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:28.8880927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp136156qj 2022-09-27T15:49:28.8881734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp136156qj/_remote_module_non_scriptable.py 2022-09-27T15:49:28.8971687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3v7wsqrf 2022-09-27T15:49:28.8972907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3v7wsqrf/_remote_module_non_scriptable.py 2022-09-27T15:49:28.8992124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpppnyu4dn 2022-09-27T15:49:28.8993552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpppnyu4dn/_remote_module_non_scriptable.py 2022-09-27T15:49:28.9409854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprtf1gryq 2022-09-27T15:49:28.9410834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprtf1gryq/_remote_module_non_scriptable.py 2022-09-27T15:49:29.1442698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:49:29.1502310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:49:29.1547564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:49:29.1972769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:49:29.8916869Z ok (2.557s) 2022-09-27T15:49:29.8917079Z 2022-09-27T15:49:29.8917741Z ---------------------------------------------------------------------- 2022-09-27T15:49:29.8918048Z Ran 1 test in 2.557s 2022-09-27T15:49:29.8918172Z 2022-09-27T15:49:29.8918226Z OK 2022-09-27T15:49:29.8918319Z 2022-09-27T15:49:29.8918413Z Generating XML reports... 2022-09-27T15:49:29.8953188Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154927.xml 2022-09-27T15:49:31.2630817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:31.2631492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:31.2640035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:31.4194491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:31.4195204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplw0byt8c 2022-09-27T15:49:31.4195974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplw0byt8c/_remote_module_non_scriptable.py 2022-09-27T15:49:31.6798203Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:49:31.6807797Z 2022-09-27T15:49:31.6807950Z Running tests... 2022-09-27T15:49:31.6808678Z ---------------------------------------------------------------------- 2022-09-27T15:49:31.9216864Z test_profiler_with_script_async_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26364 2022-09-27T15:49:31.9245599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26365 2022-09-27T15:49:31.9274086Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26366 2022-09-27T15:49:31.9302800Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26367 2022-09-27T15:49:33.0826682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:33.0827654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:33.0870575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:33.0871198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:33.1163064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:33.1163645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:33.1195349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:33.1241344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:33.1242133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:33.1242641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:33.1252856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:33.1253359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:33.1538111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:33.1538768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:33.1627184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:33.1627818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:33.2455043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qatpl2t 2022-09-27T15:49:33.2456393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qatpl2t/_remote_module_non_scriptable.py 2022-09-27T15:49:33.2517810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5lhxz5u 2022-09-27T15:49:33.2519753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5lhxz5u/_remote_module_non_scriptable.py 2022-09-27T15:49:33.2801433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpebgur9zo 2022-09-27T15:49:33.2802824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpebgur9zo/_remote_module_non_scriptable.py 2022-09-27T15:49:33.2884432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppx6_y4cf 2022-09-27T15:49:33.2885958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppx6_y4cf/_remote_module_non_scriptable.py 2022-09-27T15:49:33.5096149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:49:33.5164887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:49:33.5387954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:49:33.5648357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:49:34.2362749Z ok (2.555s) 2022-09-27T15:49:34.2362954Z 2022-09-27T15:49:34.2363268Z ---------------------------------------------------------------------- 2022-09-27T15:49:34.2363520Z Ran 1 test in 2.555s 2022-09-27T15:49:34.2363681Z 2022-09-27T15:49:34.2363760Z OK 2022-09-27T15:49:34.2363852Z 2022-09-27T15:49:34.2363946Z Generating XML reports... 2022-09-27T15:49:34.2399012Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154931.xml 2022-09-27T15:49:35.5944850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:35.5945235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:35.5953171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:35.5953632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:35.7505294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgdwu8uw0 2022-09-27T15:49:35.7506753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgdwu8uw0/_remote_module_non_scriptable.py 2022-09-27T15:49:36.0107192Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:49:36.0116944Z 2022-09-27T15:49:36.0117304Z Running tests... 2022-09-27T15:49:36.0117701Z ---------------------------------------------------------------------- 2022-09-27T15:49:36.2527416Z test_profiler_with_script_remote_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26587 2022-09-27T15:49:36.2554073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26588 2022-09-27T15:49:36.2581704Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26589 2022-09-27T15:49:36.2610042Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26590 2022-09-27T15:49:37.3942042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:37.3942660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:37.4162007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:37.4162682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:37.4298636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:37.4299165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:37.4312186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:37.4312978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:37.4431370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:37.4431956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:37.4549105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:37.4549773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:37.4810685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:37.4811323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:37.4820167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:37.4820739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:37.5602675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnc6e9xn9 2022-09-27T15:49:37.5603497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnc6e9xn9/_remote_module_non_scriptable.py 2022-09-27T15:49:37.5833754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc04sd7an 2022-09-27T15:49:37.5834762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc04sd7an/_remote_module_non_scriptable.py 2022-09-27T15:49:37.6080758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60y4ojcu 2022-09-27T15:49:37.6082303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60y4ojcu/_remote_module_non_scriptable.py 2022-09-27T15:49:37.6089685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwcrln9g 2022-09-27T15:49:37.6091983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwcrln9g/_remote_module_non_scriptable.py 2022-09-27T15:49:37.8199214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:49:37.8440748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:49:37.8656471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:49:37.8675840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:49:38.5667542Z ok (2.555s) 2022-09-27T15:49:38.5667777Z 2022-09-27T15:49:38.5668311Z ---------------------------------------------------------------------- 2022-09-27T15:49:38.5668726Z Ran 1 test in 2.555s 2022-09-27T15:49:38.5668847Z 2022-09-27T15:49:38.5668895Z OK 2022-09-27T15:49:38.5668991Z 2022-09-27T15:49:38.5669098Z Generating XML reports... 2022-09-27T15:49:38.5703340Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154936.xml 2022-09-27T15:49:39.9420474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:39.9421130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:39.9429418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:39.9429813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:40.0982996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8g1l6xkf 2022-09-27T15:49:40.0984026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8g1l6xkf/_remote_module_non_scriptable.py 2022-09-27T15:49:40.3590471Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:49:40.3600190Z 2022-09-27T15:49:40.3600300Z Running tests... 2022-09-27T15:49:40.3600746Z ---------------------------------------------------------------------- 2022-09-27T15:49:40.6087236Z test_profiler_with_script_remote_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26810 2022-09-27T15:49:40.6115109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26811 2022-09-27T15:49:40.6144239Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26812 2022-09-27T15:49:40.6181168Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26813 2022-09-27T15:49:41.7442251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:41.7442960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:41.7799263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:41.7799941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:41.7924293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:41.7924919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:41.7926591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:41.7927207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:41.7931217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:41.7932171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:41.8307427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:41.8308372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:41.8314880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:41.8315652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:41.8316371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:41.8316940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:41.9078097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3y4rj08 2022-09-27T15:49:41.9079600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3y4rj08/_remote_module_non_scriptable.py 2022-09-27T15:49:41.9582637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2z0q6yhb 2022-09-27T15:49:41.9583519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfamj4tab 2022-09-27T15:49:41.9584244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8khsbm3d 2022-09-27T15:49:41.9584967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2z0q6yhb/_remote_module_non_scriptable.py 2022-09-27T15:49:41.9585716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfamj4tab/_remote_module_non_scriptable.py 2022-09-27T15:49:41.9586465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8khsbm3d/_remote_module_non_scriptable.py 2022-09-27T15:49:42.1667882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:49:42.2112838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:49:42.2127246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:49:42.2135896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:49:42.9239048Z ok (2.563s) 2022-09-27T15:49:42.9239266Z 2022-09-27T15:49:42.9239901Z ---------------------------------------------------------------------- 2022-09-27T15:49:42.9240320Z Ran 1 test in 2.564s 2022-09-27T15:49:42.9240479Z 2022-09-27T15:49:42.9240538Z OK 2022-09-27T15:49:42.9240629Z 2022-09-27T15:49:42.9240722Z Generating XML reports... 2022-09-27T15:49:42.9275117Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154940.xml 2022-09-27T15:49:44.2823617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:44.2824265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:44.2832734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:44.4388088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:44.4388764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgrrgn2p 2022-09-27T15:49:44.4389402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgrrgn2p/_remote_module_non_scriptable.py 2022-09-27T15:49:44.7109890Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:49:44.7119814Z 2022-09-27T15:49:44.7120234Z Running tests... 2022-09-27T15:49:44.7120671Z ---------------------------------------------------------------------- 2022-09-27T15:49:44.9550589Z test_profiler_with_script_sync_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27033 2022-09-27T15:49:44.9579114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27034 2022-09-27T15:49:44.9607202Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27035 2022-09-27T15:49:44.9636815Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27036 2022-09-27T15:49:46.0601774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:46.0602461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:46.0659591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:46.0660177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:46.0752183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:46.0752840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:46.0906992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:46.0907537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:46.0977619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:46.0978142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:46.1037047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:46.1037594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:46.1130831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:46.1131348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:46.1259035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:46.1259687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:46.2233713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6597wd_ 2022-09-27T15:49:46.2234443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6597wd_/_remote_module_non_scriptable.py 2022-09-27T15:49:46.2284984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp123vr9nw 2022-09-27T15:49:46.2286566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp123vr9nw/_remote_module_non_scriptable.py 2022-09-27T15:49:46.2388848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegipjjkl 2022-09-27T15:49:46.2390865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegipjjkl/_remote_module_non_scriptable.py 2022-09-27T15:49:46.2508500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qdg1jnk 2022-09-27T15:49:46.2509972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qdg1jnk/_remote_module_non_scriptable.py 2022-09-27T15:49:46.4782165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:49:46.4829739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:49:46.4950873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:49:46.5063401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:49:47.1693089Z ok (2.457s) 2022-09-27T15:49:47.1693322Z 2022-09-27T15:49:47.1693639Z ---------------------------------------------------------------------- 2022-09-27T15:49:47.1693889Z Ran 1 test in 2.457s 2022-09-27T15:49:47.1694008Z 2022-09-27T15:49:47.1694061Z OK 2022-09-27T15:49:47.1694154Z 2022-09-27T15:49:47.1694247Z Generating XML reports... 2022-09-27T15:49:47.1730924Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154944.xml 2022-09-27T15:49:48.5215789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:48.5216413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:48.5224451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:48.5224833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:48.6756735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_xilche 2022-09-27T15:49:48.6757828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_xilche/_remote_module_non_scriptable.py 2022-09-27T15:49:48.9352833Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:49:48.9362724Z 2022-09-27T15:49:48.9363032Z Running tests... 2022-09-27T15:49:48.9363673Z ---------------------------------------------------------------------- 2022-09-27T15:49:49.1798868Z test_profiler_with_script_sync_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27256 2022-09-27T15:49:49.1826613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27257 2022-09-27T15:49:49.1853945Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27258 2022-09-27T15:49:49.1883560Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27259 2022-09-27T15:49:50.2903311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:50.2904043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:50.2905000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:50.2905622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:50.2951636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:50.2952289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:50.2987787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:50.2988312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:50.3299110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:50.3299867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:50.3304685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:50.3305442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:50.3344069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:50.3344710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:50.3373553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:50.3374124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:50.4550736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfter104d 2022-09-27T15:49:50.4551600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfter104d/_remote_module_non_scriptable.py 2022-09-27T15:49:50.4560944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp430xmoe6 2022-09-27T15:49:50.4563147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp430xmoe6/_remote_module_non_scriptable.py 2022-09-27T15:49:50.4602042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcja578bq 2022-09-27T15:49:50.4604392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcja578bq/_remote_module_non_scriptable.py 2022-09-27T15:49:50.4624881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_fy77tp 2022-09-27T15:49:50.4626754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_fy77tp/_remote_module_non_scriptable.py 2022-09-27T15:49:50.7087778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:49:50.7119490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:49:50.7183258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:49:50.7183859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:49:51.3938987Z ok (2.457s) 2022-09-27T15:49:51.3939205Z 2022-09-27T15:49:51.3939632Z ---------------------------------------------------------------------- 2022-09-27T15:49:51.3939979Z Ran 1 test in 2.458s 2022-09-27T15:49:51.3940104Z 2022-09-27T15:49:51.3940151Z OK 2022-09-27T15:49:51.3940283Z 2022-09-27T15:49:51.3940424Z Generating XML reports... 2022-09-27T15:49:51.3977281Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154948.xml 2022-09-27T15:49:52.7579405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:52.7579788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:52.7587416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:52.7587789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:52.9126972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgpuj8ii 2022-09-27T15:49:52.9127744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgpuj8ii/_remote_module_non_scriptable.py 2022-09-27T15:49:53.1750522Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:49:53.1760308Z 2022-09-27T15:49:53.1760644Z Running tests... 2022-09-27T15:49:53.1761252Z ---------------------------------------------------------------------- 2022-09-27T15:49:53.4145336Z test_profiler_with_sync_rpc_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27479 2022-09-27T15:49:53.4173084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27480 2022-09-27T15:49:53.4201428Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27481 2022-09-27T15:49:53.4229983Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27482 2022-09-27T15:49:54.5163303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:54.5163939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:54.5331444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:54.5332120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:54.5449791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:54.5450287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:54.5487643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:54.5488323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:54.5531683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:54.5532356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:54.5716208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:54.5716752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:54.5831133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:54.5831924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:54.5862198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:54.5862802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:54.6805276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7tjlz05x 2022-09-27T15:49:54.6806076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7tjlz05x/_remote_module_non_scriptable.py 2022-09-27T15:49:54.6977267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa966iknw 2022-09-27T15:49:54.6977939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa966iknw/_remote_module_non_scriptable.py 2022-09-27T15:49:54.7097376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0e7_rsc 2022-09-27T15:49:54.7098103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0e7_rsc/_remote_module_non_scriptable.py 2022-09-27T15:49:54.7122497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hmcbbla 2022-09-27T15:49:54.7125956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hmcbbla/_remote_module_non_scriptable.py 2022-09-27T15:49:54.9398231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:49:54.9549937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:49:54.9648416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:49:54.9697716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:49:55.6286748Z ok (2.452s) 2022-09-27T15:49:55.6287011Z 2022-09-27T15:49:55.6287490Z ---------------------------------------------------------------------- 2022-09-27T15:49:55.6287742Z Ran 1 test in 2.453s 2022-09-27T15:49:55.6287861Z 2022-09-27T15:49:55.6287921Z OK 2022-09-27T15:49:55.6288015Z 2022-09-27T15:49:55.6288108Z Generating XML reports... 2022-09-27T15:49:55.6322454Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154953.xml 2022-09-27T15:49:56.9821676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:56.9822116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:56.9829462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:56.9829862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:57.1403294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpesqzq5cy 2022-09-27T15:49:57.1404226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpesqzq5cy/_remote_module_non_scriptable.py 2022-09-27T15:49:57.4003122Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:49:57.4012504Z 2022-09-27T15:49:57.4012622Z Running tests... 2022-09-27T15:49:57.4013010Z ---------------------------------------------------------------------- 2022-09-27T15:49:57.6413163Z test_profiler_with_sync_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27702 2022-09-27T15:49:57.6440850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27703 2022-09-27T15:49:57.6467890Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27704 2022-09-27T15:49:57.6497925Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27705 2022-09-27T15:49:58.7597688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:58.7598272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:58.7642019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:58.7642686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:58.7643677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:58.7644290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:58.7665644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:49:58.7666283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:49:58.7985603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:58.7986336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:58.8030067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:58.8030706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:58.8039319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:58.8039886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:58.8053938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:49:58.8054511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:49:58.9276987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7l0zunhz 2022-09-27T15:49:58.9278096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7l0zunhz/_remote_module_non_scriptable.py 2022-09-27T15:49:58.9302933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqry25_9n 2022-09-27T15:49:58.9304923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqry25_9n/_remote_module_non_scriptable.py 2022-09-27T15:49:58.9307014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53jue4fh 2022-09-27T15:49:58.9309369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53jue4fh/_remote_module_non_scriptable.py 2022-09-27T15:49:58.9324499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8c9wk6i 2022-09-27T15:49:58.9326055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8c9wk6i/_remote_module_non_scriptable.py 2022-09-27T15:49:59.1851251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:49:59.1861865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:49:59.1884353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:49:59.1907746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:49:59.8554887Z ok (2.454s) 2022-09-27T15:49:59.8555202Z 2022-09-27T15:49:59.8555672Z ---------------------------------------------------------------------- 2022-09-27T15:49:59.8555995Z Ran 1 test in 2.454s 2022-09-27T15:49:59.8556356Z 2022-09-27T15:49:59.8556418Z OK 2022-09-27T15:49:59.8556499Z 2022-09-27T15:49:59.8556593Z Generating XML reports... 2022-09-27T15:49:59.8590799Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154957.xml 2022-09-27T15:50:01.2216989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:01.2217602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:01.2226651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:01.2227236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:01.3864602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2rwo6l9 2022-09-27T15:50:01.3865505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2rwo6l9/_remote_module_non_scriptable.py 2022-09-27T15:50:01.6475144Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:50:01.6485882Z 2022-09-27T15:50:01.6486379Z Running tests... 2022-09-27T15:50:01.6486790Z ---------------------------------------------------------------------- 2022-09-27T15:50:01.9031502Z test_profiler_with_sync_rpc_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27925 2022-09-27T15:50:01.9060080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27926 2022-09-27T15:50:01.9091213Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27927 2022-09-27T15:50:01.9122643Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27928 2022-09-27T15:50:02.9970653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:02.9971301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:03.0084122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:03.0084785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:03.0305969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:03.0306648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:03.0346310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:03.0346990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:03.0460729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:03.0461269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:03.0682577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:03.0683212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:03.0849247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:03.0849861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:03.1203338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:03.1204160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:03.1613878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpouk42752 2022-09-27T15:50:03.1614579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpouk42752/_remote_module_non_scriptable.py 2022-09-27T15:50:03.1733984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcmwc85u 2022-09-27T15:50:03.1734840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcmwc85u/_remote_module_non_scriptable.py 2022-09-27T15:50:03.1945153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgwvitg4 2022-09-27T15:50:03.1945848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgwvitg4/_remote_module_non_scriptable.py 2022-09-27T15:50:03.2468690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6u91m8n 2022-09-27T15:50:03.2469523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6u91m8n/_remote_module_non_scriptable.py 2022-09-27T15:50:03.4176876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:50:03.4488825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:50:03.4521891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:50:03.5069771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:50:06.1208679Z ok (4.472s) 2022-09-27T15:50:06.1208973Z 2022-09-27T15:50:06.1209375Z ---------------------------------------------------------------------- 2022-09-27T15:50:06.1209612Z Ran 1 test in 4.472s 2022-09-27T15:50:06.1209725Z 2022-09-27T15:50:06.1209785Z OK 2022-09-27T15:50:06.1209876Z 2022-09-27T15:50:06.1209966Z Generating XML reports... 2022-09-27T15:50:06.1244982Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155001.xml 2022-09-27T15:50:07.5087447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:07.5088085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:07.5096871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:07.5097529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:07.6695599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4j08_tcj 2022-09-27T15:50:07.6696277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4j08_tcj/_remote_module_non_scriptable.py 2022-09-27T15:50:07.9336431Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:50:07.9346334Z 2022-09-27T15:50:07.9346692Z Running tests... 2022-09-27T15:50:07.9347304Z ---------------------------------------------------------------------- 2022-09-27T15:50:08.1779702Z test_profiler_with_sync_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28148 2022-09-27T15:50:08.1807367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28149 2022-09-27T15:50:08.1835257Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28150 2022-09-27T15:50:08.1864406Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28151 2022-09-27T15:50:09.2814961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:09.2815609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:09.3150095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:09.3150779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:09.3161661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:09.3162304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:09.3316553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:09.3317168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:09.3521349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:09.3522050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:09.3541136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:09.3541767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:09.3693928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:09.3694711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:09.3887129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:09.3887802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:09.4423645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mpa4i4h 2022-09-27T15:50:09.4424820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mpa4i4h/_remote_module_non_scriptable.py 2022-09-27T15:50:09.4828866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4gvxn79t 2022-09-27T15:50:09.4829711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4gvxn79t/_remote_module_non_scriptable.py 2022-09-27T15:50:09.4984506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1cnvvx2 2022-09-27T15:50:09.4985456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1cnvvx2/_remote_module_non_scriptable.py 2022-09-27T15:50:09.5170177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5qoo8rq 2022-09-27T15:50:09.5171429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5qoo8rq/_remote_module_non_scriptable.py 2022-09-27T15:50:09.7494854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:50:09.7566456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:50:09.7610280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:50:09.8294665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:50:12.3952730Z ok (4.460s) 2022-09-27T15:50:12.3952978Z 2022-09-27T15:50:12.3953387Z ---------------------------------------------------------------------- 2022-09-27T15:50:12.3953760Z Ran 1 test in 4.461s 2022-09-27T15:50:12.3953955Z 2022-09-27T15:50:12.3954046Z OK 2022-09-27T15:50:12.3954186Z 2022-09-27T15:50:12.3954333Z Generating XML reports... 2022-09-27T15:50:12.3990518Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155007.xml 2022-09-27T15:50:13.8037330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:13.8037725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:13.8046018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:13.8046423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:13.9629899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5yahmad 2022-09-27T15:50:13.9630753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5yahmad/_remote_module_non_scriptable.py 2022-09-27T15:50:14.2274129Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:50:14.2284313Z 2022-09-27T15:50:14.2284432Z Running tests... 2022-09-27T15:50:14.2285726Z ---------------------------------------------------------------------- 2022-09-27T15:50:14.4751531Z test_py_built_in (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28371 2022-09-27T15:50:14.4779394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28372 2022-09-27T15:50:14.4808697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28373 2022-09-27T15:50:14.4838277Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28374 2022-09-27T15:50:15.6207756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:15.6208392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:15.6561670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:15.6562264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:15.6571334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:15.6572023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:15.6802352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:15.6802892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:15.6918485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:15.6919095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:15.6972493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:15.6973107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:15.7171481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:15.7172130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:15.7337467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:15.7338114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:15.7843813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvj_71er 2022-09-27T15:50:15.7844465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvj_71er/_remote_module_non_scriptable.py 2022-09-27T15:50:15.8190483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeubzrz6y 2022-09-27T15:50:15.8191333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeubzrz6y/_remote_module_non_scriptable.py 2022-09-27T15:50:15.8575406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0c0ct8h 2022-09-27T15:50:15.8576150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0c0ct8h/_remote_module_non_scriptable.py 2022-09-27T15:50:15.8608895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc19cme24 2022-09-27T15:50:15.8610298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc19cme24/_remote_module_non_scriptable.py 2022-09-27T15:50:16.0445870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:50:16.0792326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:50:16.1171858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:50:16.1180446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:50:16.7896496Z ok (2.561s) 2022-09-27T15:50:16.7896778Z 2022-09-27T15:50:16.7897487Z ---------------------------------------------------------------------- 2022-09-27T15:50:16.7897732Z Ran 1 test in 2.561s 2022-09-27T15:50:16.7897930Z 2022-09-27T15:50:16.7897993Z OK 2022-09-27T15:50:16.7898265Z 2022-09-27T15:50:16.7898524Z Generating XML reports... 2022-09-27T15:50:16.7935729Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155014.xml 2022-09-27T15:50:18.1895111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:18.1895488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:18.1902945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:18.1903469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:18.3487122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg493kusf 2022-09-27T15:50:18.3488101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg493kusf/_remote_module_non_scriptable.py 2022-09-27T15:50:18.6098049Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:50:18.6108468Z 2022-09-27T15:50:18.6108887Z Running tests... 2022-09-27T15:50:18.6109282Z ---------------------------------------------------------------------- 2022-09-27T15:50:18.8583292Z test_py_class_constructor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28594 2022-09-27T15:50:18.8610779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28595 2022-09-27T15:50:18.8638915Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28596 2022-09-27T15:50:18.8669478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28597 2022-09-27T15:50:19.9957577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:19.9958281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:20.0317445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:20.0318046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:20.0353647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:20.0354255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:20.0708828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:20.0709494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:20.0978519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:20.0979186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:20.1066016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:20.1066642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:20.1344738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:20.1345415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:20.1444670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:20.1445316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:20.1614290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp121u3akw 2022-09-27T15:50:20.1615442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp121u3akw/_remote_module_non_scriptable.py 2022-09-27T15:50:20.1983929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1hg6tp0 2022-09-27T15:50:20.1984708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1hg6tp0/_remote_module_non_scriptable.py 2022-09-27T15:50:20.2732902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7eil2yva 2022-09-27T15:50:20.2734850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7eil2yva/_remote_module_non_scriptable.py 2022-09-27T15:50:20.2850424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpus7z5pub 2022-09-27T15:50:20.2851436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpus7z5pub/_remote_module_non_scriptable.py 2022-09-27T15:50:20.4424653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:50:20.4752323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:50:20.5447231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:50:20.5458027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:50:21.1726178Z ok (2.561s) 2022-09-27T15:50:21.1726393Z 2022-09-27T15:50:21.1726801Z ---------------------------------------------------------------------- 2022-09-27T15:50:21.1727197Z Ran 1 test in 2.562s 2022-09-27T15:50:21.1727385Z 2022-09-27T15:50:21.1727497Z OK 2022-09-27T15:50:21.1727648Z 2022-09-27T15:50:21.1727789Z Generating XML reports... 2022-09-27T15:50:21.1763350Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155018.xml 2022-09-27T15:50:22.5712081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:22.5712692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:22.5721383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:22.5722016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:22.7294846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpun30vmfn 2022-09-27T15:50:22.7295861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpun30vmfn/_remote_module_non_scriptable.py 2022-09-27T15:50:22.9948662Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:50:22.9958523Z 2022-09-27T15:50:22.9958767Z Running tests... 2022-09-27T15:50:22.9959190Z ---------------------------------------------------------------------- 2022-09-27T15:50:23.2478179Z test_py_class_instance_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28817 2022-09-27T15:50:23.2505152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28818 2022-09-27T15:50:23.2532801Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28819 2022-09-27T15:50:23.2562762Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28820 2022-09-27T15:50:24.3897836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:24.3898487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:24.4018174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:24.4018758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:24.4120717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:24.4121365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:24.4122447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:24.4123194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:24.4285355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:24.4285903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:24.4409010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:24.4409680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:24.4500963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:24.4501678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:24.4506519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:24.4507125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:24.5550422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlf872qk 2022-09-27T15:50:24.5551202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlf872qk/_remote_module_non_scriptable.py 2022-09-27T15:50:24.5690811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnf2cuiv 2022-09-27T15:50:24.5691969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnf2cuiv/_remote_module_non_scriptable.py 2022-09-27T15:50:24.5777759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzrmu3e7n 2022-09-27T15:50:24.5778526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5pre_cjg 2022-09-27T15:50:24.5779527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzrmu3e7n/_remote_module_non_scriptable.py 2022-09-27T15:50:24.5780191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5pre_cjg/_remote_module_non_scriptable.py 2022-09-27T15:50:24.8128843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:50:24.8303562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:50:24.8365413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:50:24.8365963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:50:25.4619189Z ok (2.466s) 2022-09-27T15:50:25.4619431Z 2022-09-27T15:50:25.4619757Z ---------------------------------------------------------------------- 2022-09-27T15:50:25.4620037Z Ran 1 test in 2.466s 2022-09-27T15:50:25.4620156Z 2022-09-27T15:50:25.4620221Z OK 2022-09-27T15:50:25.4620313Z 2022-09-27T15:50:25.4620419Z Generating XML reports... 2022-09-27T15:50:25.4656865Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155022.xml 2022-09-27T15:50:26.8709383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:26.8709776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:26.8717904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:26.8718277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:27.0325072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcacs2xep 2022-09-27T15:50:27.0326250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcacs2xep/_remote_module_non_scriptable.py 2022-09-27T15:50:27.2959934Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:50:27.2969584Z 2022-09-27T15:50:27.2969952Z Running tests... 2022-09-27T15:50:27.2970521Z ---------------------------------------------------------------------- 2022-09-27T15:50:27.5497885Z test_py_class_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29040 2022-09-27T15:50:27.5525611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29041 2022-09-27T15:50:27.5553206Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29042 2022-09-27T15:50:27.5583601Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29043 2022-09-27T15:50:28.6598343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:28.6599032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:28.6740403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:28.6740980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:28.6898098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:28.6898750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:28.6978697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:28.6979340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:28.7046219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:28.7046857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:28.7131074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:28.7283000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:28.7283950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:28.7284593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:28.7420733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:28.7421411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:28.8252016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyw2n2ety 2022-09-27T15:50:28.8252839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyw2n2ety/_remote_module_non_scriptable.py 2022-09-27T15:50:28.8409637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmipgez_ 2022-09-27T15:50:28.8410502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmipgez_/_remote_module_non_scriptable.py 2022-09-27T15:50:28.8565670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79t_qs84 2022-09-27T15:50:28.8566549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79t_qs84/_remote_module_non_scriptable.py 2022-09-27T15:50:28.8692154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7bb2t_xz 2022-09-27T15:50:28.8693469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7bb2t_xz/_remote_module_non_scriptable.py 2022-09-27T15:50:29.0842306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:50:29.1001121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:50:29.1165744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:50:29.1278351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:50:29.7641769Z ok (2.467s) 2022-09-27T15:50:29.7642032Z 2022-09-27T15:50:29.7642376Z ---------------------------------------------------------------------- 2022-09-27T15:50:29.7642899Z Ran 1 test in 2.467s 2022-09-27T15:50:29.7643016Z 2022-09-27T15:50:29.7643075Z OK 2022-09-27T15:50:29.7643155Z 2022-09-27T15:50:29.7643248Z Generating XML reports... 2022-09-27T15:50:29.7677653Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155027.xml 2022-09-27T15:50:31.1505582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:31.1505973Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:31.1513764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:31.1514319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:31.3060902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mph934u 2022-09-27T15:50:31.3061569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mph934u/_remote_module_non_scriptable.py 2022-09-27T15:50:31.5764301Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:50:31.5773423Z 2022-09-27T15:50:31.5773524Z Running tests... 2022-09-27T15:50:31.5774122Z ---------------------------------------------------------------------- 2022-09-27T15:50:31.8202648Z test_py_class_static_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29263 2022-09-27T15:50:31.8229607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29264 2022-09-27T15:50:31.8257314Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29265 2022-09-27T15:50:31.8287039Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29266 2022-09-27T15:50:32.9549928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:32.9550568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:32.9702256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:32.9702939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:32.9733373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:32.9734011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:32.9804188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:32.9804763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:32.9924322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:32.9924873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:33.0093987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:33.0094765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:33.0130456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:33.0131236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:33.0194931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:33.0195595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:33.1197128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxq3sm9_s 2022-09-27T15:50:33.1197847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxq3sm9_s/_remote_module_non_scriptable.py 2022-09-27T15:50:33.1375172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79bc6zja 2022-09-27T15:50:33.1376333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79bc6zja/_remote_module_non_scriptable.py 2022-09-27T15:50:33.1405302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppp7yr09a 2022-09-27T15:50:33.1407214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppp7yr09a/_remote_module_non_scriptable.py 2022-09-27T15:50:33.1468783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkxu4w86 2022-09-27T15:50:33.1471177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkxu4w86/_remote_module_non_scriptable.py 2022-09-27T15:50:33.3747594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:50:33.3938191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:50:33.3986745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:50:33.4071198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:50:34.0343775Z ok (2.457s) 2022-09-27T15:50:34.0343993Z 2022-09-27T15:50:34.0344440Z ---------------------------------------------------------------------- 2022-09-27T15:50:34.0344851Z Ran 1 test in 2.457s 2022-09-27T15:50:34.0345031Z 2022-09-27T15:50:34.0345121Z OK 2022-09-27T15:50:34.0345267Z 2022-09-27T15:50:34.0345403Z Generating XML reports... 2022-09-27T15:50:34.0380982Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155031.xml 2022-09-27T15:50:35.4609300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:35.4609682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:35.4617287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:35.4617655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:35.6233724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvb_6vqyw 2022-09-27T15:50:35.6234267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvb_6vqyw/_remote_module_non_scriptable.py 2022-09-27T15:50:35.8872369Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:50:35.8882611Z 2022-09-27T15:50:35.8882703Z Running tests... 2022-09-27T15:50:35.8883226Z ---------------------------------------------------------------------- 2022-09-27T15:50:36.1302471Z test_py_function_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29486 2022-09-27T15:50:36.1329724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29487 2022-09-27T15:50:36.1358059Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29488 2022-09-27T15:50:36.1389134Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29489 2022-09-27T15:50:37.2648886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:37.2649554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:37.2698438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:37.2699102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:37.2703649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:37.2704269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:37.3045720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:37.3046336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:37.3095703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:37.3096310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:37.3103420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:37.3104020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:37.3177363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:37.3177978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:37.3559335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:37.3560010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:37.4318010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmtkpj6yr 2022-09-27T15:50:37.4318779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmtkpj6yr/_remote_module_non_scriptable.py 2022-09-27T15:50:37.4356586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperiq5k8x 2022-09-27T15:50:37.4358107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperiq5k8x/_remote_module_non_scriptable.py 2022-09-27T15:50:37.4370502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ihf89w2 2022-09-27T15:50:37.4372444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ihf89w2/_remote_module_non_scriptable.py 2022-09-27T15:50:37.4825950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeyryqmfu 2022-09-27T15:50:37.4826763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeyryqmfu/_remote_module_non_scriptable.py 2022-09-27T15:50:37.6882520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:50:37.6958103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:50:37.7115231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:50:37.7441792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:50:37.9386297Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:50:37.9387287Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-09-27T15:50:37.9387837Z Traceback (most recent call last): 2022-09-27T15:50:37.9388732Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:50:37.9389443Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:50:37.9390063Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-09-27T15:50:37.9433337Z 2022-09-27T15:50:37.9542351Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:50:37.9543293Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-09-27T15:50:37.9543841Z Traceback (most recent call last): 2022-09-27T15:50:37.9544711Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:50:37.9545417Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:50:37.9546050Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-09-27T15:50:37.9546645Z 2022-09-27T15:50:37.9582264Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:50:37.9583246Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-09-27T15:50:37.9583675Z Traceback (most recent call last): 2022-09-27T15:50:37.9584454Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:50:37.9585025Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:50:37.9598687Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-09-27T15:50:37.9599004Z 2022-09-27T15:50:37.9644986Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:50:37.9645834Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-09-27T15:50:37.9646395Z Traceback (most recent call last): 2022-09-27T15:50:37.9647295Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:50:37.9648007Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:50:37.9648646Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-09-27T15:50:37.9648999Z 2022-09-27T15:50:38.3444948Z ok (2.456s) 2022-09-27T15:50:38.3445223Z 2022-09-27T15:50:38.3445723Z ---------------------------------------------------------------------- 2022-09-27T15:50:38.3446053Z Ran 1 test in 2.456s 2022-09-27T15:50:38.3446169Z 2022-09-27T15:50:38.3446228Z OK 2022-09-27T15:50:38.3446318Z 2022-09-27T15:50:38.3446412Z Generating XML reports... 2022-09-27T15:50:38.3481022Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155035.xml 2022-09-27T15:50:39.7625284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:39.7625945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:39.7635629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:39.7636255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:39.9229016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcxsf_vak 2022-09-27T15:50:39.9229896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcxsf_vak/_remote_module_non_scriptable.py 2022-09-27T15:50:40.1825059Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:50:40.1835499Z 2022-09-27T15:50:40.4308950Z Running tests... 2022-09-27T15:50:40.4309642Z ---------------------------------------------------------------------- 2022-09-27T15:50:40.4310309Z test_py_multi_async_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29709 2022-09-27T15:50:40.4339726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29710 2022-09-27T15:50:40.4368988Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29711 2022-09-27T15:50:40.4400231Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29712 2022-09-27T15:50:41.5338233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:41.5338841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:41.5574760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:41.5575420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:41.5698248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:41.5698790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:41.5742529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:41.5743383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:41.5958588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:41.5959461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:41.5960193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:41.5960739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:41.6119037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:41.6119708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:41.6321495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:41.6322133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:41.6962405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoo6e72rk 2022-09-27T15:50:41.6964821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoo6e72rk/_remote_module_non_scriptable.py 2022-09-27T15:50:41.7387629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggp3tfwh 2022-09-27T15:50:41.7388449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggp3tfwh/_remote_module_non_scriptable.py 2022-09-27T15:50:41.7402813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqpbchtz 2022-09-27T15:50:41.7404609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqpbchtz/_remote_module_non_scriptable.py 2022-09-27T15:50:41.7596273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphh45ptae 2022-09-27T15:50:41.7596963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphh45ptae/_remote_module_non_scriptable.py 2022-09-27T15:50:41.9516292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:50:41.9976037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:50:41.9999419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:50:42.0195296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:50:42.6455477Z ok (2.462s) 2022-09-27T15:50:42.6455904Z 2022-09-27T15:50:42.6456670Z ---------------------------------------------------------------------- 2022-09-27T15:50:42.6457146Z Ran 1 test in 2.462s 2022-09-27T15:50:42.6457350Z 2022-09-27T15:50:42.6457478Z OK 2022-09-27T15:50:42.6457627Z 2022-09-27T15:50:42.6457728Z Generating XML reports... 2022-09-27T15:50:42.6492045Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155040.xml 2022-09-27T15:50:44.0152139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:44.0152551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:44.0161418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:44.0161783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:44.1782257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqamk0op 2022-09-27T15:50:44.1791884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqamk0op/_remote_module_non_scriptable.py 2022-09-27T15:50:44.4465567Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:50:44.4476040Z 2022-09-27T15:50:44.4476160Z Running tests... 2022-09-27T15:50:44.4476997Z ---------------------------------------------------------------------- 2022-09-27T15:50:44.7032730Z test_py_nested_pickle (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29932 2022-09-27T15:50:44.7061788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29933 2022-09-27T15:50:44.7091543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29934 2022-09-27T15:50:44.7123877Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29935 2022-09-27T15:50:45.8512838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:45.8513543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:45.8675718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:45.8676277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:45.8845543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:45.8846133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:45.8898565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:45.8899104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:45.9061610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:45.9062235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:45.9105894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:45.9106491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:45.9217563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:45.9218250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:45.9473954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:46.0164367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:46.0165035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvav5kbo 2022-09-27T15:50:46.0165795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvav5kbo/_remote_module_non_scriptable.py 2022-09-27T15:50:46.0329813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6yciq1a1 2022-09-27T15:50:46.0330861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6yciq1a1/_remote_module_non_scriptable.py 2022-09-27T15:50:46.0483045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wl3yz0g 2022-09-27T15:50:46.0484335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wl3yz0g/_remote_module_non_scriptable.py 2022-09-27T15:50:46.0738409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1zlkzwij 2022-09-27T15:50:46.0740932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1zlkzwij/_remote_module_non_scriptable.py 2022-09-27T15:50:46.2720753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:50:46.2899727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:50:46.3049530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:50:46.3361370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:50:47.0182148Z ok (2.570s) 2022-09-27T15:50:47.0182403Z 2022-09-27T15:50:47.0182866Z ---------------------------------------------------------------------- 2022-09-27T15:50:47.0183353Z Ran 1 test in 2.570s 2022-09-27T15:50:47.0183460Z 2022-09-27T15:50:47.0183520Z OK 2022-09-27T15:50:47.0183613Z 2022-09-27T15:50:47.0183775Z Generating XML reports... 2022-09-27T15:50:47.0217528Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155044.xml 2022-09-27T15:50:48.3879912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:48.3880293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:48.3888581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:48.3888930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:48.5471168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmlzb873u 2022-09-27T15:50:48.5472429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmlzb873u/_remote_module_non_scriptable.py 2022-09-27T15:50:48.8096513Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:50:48.8106254Z 2022-09-27T15:50:48.8106398Z Running tests... 2022-09-27T15:50:48.8106811Z ---------------------------------------------------------------------- 2022-09-27T15:50:49.0615600Z test_py_no_return_result (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30155 2022-09-27T15:50:49.0643590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30156 2022-09-27T15:50:49.0671386Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30157 2022-09-27T15:50:49.0702457Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30158 2022-09-27T15:50:50.1788218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:50.1788919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:50.2066097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:50.2066741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:50.2140487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:50.2141126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:50.2235942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:50.2236535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:50.2340176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:50.2340776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:50.2451384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:50.2451940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:50.2624139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:50.2624761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:50.2723324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:50.2723946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:50.3416437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_ohs6b9 2022-09-27T15:50:50.3417201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_ohs6b9/_remote_module_non_scriptable.py 2022-09-27T15:50:50.3755728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlhytz8z 2022-09-27T15:50:50.3756664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlhytz8z/_remote_module_non_scriptable.py 2022-09-27T15:50:50.3910551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplb2etcjk 2022-09-27T15:50:50.3911949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplb2etcjk/_remote_module_non_scriptable.py 2022-09-27T15:50:50.4000677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcwk1vgl1 2022-09-27T15:50:50.4001970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcwk1vgl1/_remote_module_non_scriptable.py 2022-09-27T15:50:50.6029074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:50:50.6383569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:50:50.6508591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:50:50.6608720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:50:50.8540660Z do nothing 2022-09-27T15:50:50.8543057Z do nothing 2022-09-27T15:50:50.8740592Z do nothing 2022-09-27T15:50:50.8742521Z do nothing 2022-09-27T15:50:50.8823061Z do nothing 2022-09-27T15:50:50.8836849Z do nothing 2022-09-27T15:50:50.8852334Z do nothing 2022-09-27T15:50:50.8855794Z do nothing 2022-09-27T15:50:51.2759135Z ok (2.465s) 2022-09-27T15:50:51.2759453Z 2022-09-27T15:50:51.2759973Z ---------------------------------------------------------------------- 2022-09-27T15:50:51.2760234Z Ran 1 test in 2.465s 2022-09-27T15:50:51.2760353Z 2022-09-27T15:50:51.2760415Z OK 2022-09-27T15:50:51.2760494Z 2022-09-27T15:50:51.2760589Z Generating XML reports... 2022-09-27T15:50:51.2796061Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155048.xml 2022-09-27T15:50:52.6665471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:52.6665857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:52.6675565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:52.6675924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:52.8310604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgmti5kr 2022-09-27T15:50:52.8311651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgmti5kr/_remote_module_non_scriptable.py 2022-09-27T15:50:53.0922598Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:50:53.0931361Z 2022-09-27T15:50:53.0931775Z Running tests... 2022-09-27T15:50:53.0932174Z ---------------------------------------------------------------------- 2022-09-27T15:50:53.3367685Z test_py_raise_in_user_func (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30378 2022-09-27T15:50:53.3395989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30379 2022-09-27T15:50:53.3423848Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30380 2022-09-27T15:50:53.3454889Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30381 2022-09-27T15:50:54.4278305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:54.4278951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:54.4594036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:54.4594966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:54.4599127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:54.4599890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:54.4636907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:54.4637567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:54.4826265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:54.4826933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:54.4965803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:54.4966342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:54.4978570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:54.4979210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:54.5175105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:54.5175781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:54.5903493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5yxa7n63 2022-09-27T15:50:54.5904300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5yxa7n63/_remote_module_non_scriptable.py 2022-09-27T15:50:54.6234264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyvt6rjp 2022-09-27T15:50:54.6235606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyvt6rjp/_remote_module_non_scriptable.py 2022-09-27T15:50:54.6242684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2d7xxwsu 2022-09-27T15:50:54.6245419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2d7xxwsu/_remote_module_non_scriptable.py 2022-09-27T15:50:54.6428124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6z6mnzyg 2022-09-27T15:50:54.6429291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6z6mnzyg/_remote_module_non_scriptable.py 2022-09-27T15:50:54.8437561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:50:54.8769402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:50:54.8812392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:50:54.8958445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:50:55.1647777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:50:55.1751037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:50:55.1851876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:50:55.1857214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:50:55.1858524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:50:55.1861349Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:50:55.1864274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:50:55.1865780Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:50:55.6512022Z ok (2.558s) 2022-09-27T15:50:55.6512294Z 2022-09-27T15:50:55.6512665Z ---------------------------------------------------------------------- 2022-09-27T15:50:55.6513001Z Ran 1 test in 2.558s 2022-09-27T15:50:55.6513106Z 2022-09-27T15:50:55.6513168Z OK 2022-09-27T15:50:55.6513262Z 2022-09-27T15:50:55.6513359Z Generating XML reports... 2022-09-27T15:50:55.6550104Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155053.xml 2022-09-27T15:50:57.0156431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:57.0156822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:57.0165651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:57.0166125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:57.1770470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffhr_98r 2022-09-27T15:50:57.1771437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffhr_98r/_remote_module_non_scriptable.py 2022-09-27T15:50:57.4417387Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:50:57.4426920Z 2022-09-27T15:50:57.4427024Z Running tests... 2022-09-27T15:50:57.4427598Z ---------------------------------------------------------------------- 2022-09-27T15:50:57.6839343Z test_py_raise_in_user_func_escaped_str (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30613 2022-09-27T15:50:57.6867261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30614 2022-09-27T15:50:57.6895184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30615 2022-09-27T15:50:57.6925048Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30616 2022-09-27T15:50:58.7788046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:58.7788736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:58.8086282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:58.8086999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:58.8152056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:58.8152644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:58.8166254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:58.8166773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:58.8219003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:50:58.8219655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:50:58.8464686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:58.8465352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:58.8550826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:58.8551495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:58.8604115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:50:58.8604745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:50:58.9424106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mkyh6uf 2022-09-27T15:50:58.9424939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mkyh6uf/_remote_module_non_scriptable.py 2022-09-27T15:50:58.9719917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphgewxr_v 2022-09-27T15:50:58.9722292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphgewxr_v/_remote_module_non_scriptable.py 2022-09-27T15:50:58.9807447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmb8u5pf 2022-09-27T15:50:58.9808959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmb8u5pf/_remote_module_non_scriptable.py 2022-09-27T15:50:58.9862755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwmg7v6d 2022-09-27T15:50:58.9864784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwmg7v6d/_remote_module_non_scriptable.py 2022-09-27T15:50:59.1977113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:50:59.2273670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:50:59.2357301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:50:59.2433764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:50:59.4879197Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:50:59.4880199Z ValueError('\nFirst line of error \n next line of error \n last line of error') 2022-09-27T15:50:59.4880785Z Traceback (most recent call last): 2022-09-27T15:50:59.4881663Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:50:59.4882376Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:50:59.4883365Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func_escape 2022-09-27T15:50:59.4884074Z raise ValueError(expected_err_escape) 2022-09-27T15:50:59.4884486Z ValueError: 2022-09-27T15:50:59.4884864Z First line of error 2022-09-27T15:50:59.4885254Z next line of error 2022-09-27T15:50:59.4885614Z last line of error 2022-09-27T15:50:59.4885838Z 2022-09-27T15:50:59.4964277Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:50:59.4965183Z ValueError('\nFirst line of error \n next line of error \n last line of error') 2022-09-27T15:50:59.4965757Z Traceback (most recent call last): 2022-09-27T15:50:59.4966614Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:50:59.4967323Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:50:59.4968316Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func_escape 2022-09-27T15:50:59.4969155Z raise ValueError(expected_err_escape) 2022-09-27T15:50:59.4969583Z ValueError: 2022-09-27T15:50:59.4969961Z First line of error 2022-09-27T15:50:59.4970355Z next line of error 2022-09-27T15:50:59.4970717Z last line of error 2022-09-27T15:50:59.4970950Z 2022-09-27T15:50:59.5027318Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:50:59.5028229Z ValueError('\nFirst line of error \n next line of error \n last line of error') 2022-09-27T15:50:59.5028780Z Traceback (most recent call last): 2022-09-27T15:50:59.5029647Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:50:59.5030360Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:50:59.5031405Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func_escape 2022-09-27T15:50:59.5032077Z raise ValueError(expected_err_escape) 2022-09-27T15:50:59.5032502Z ValueError: 2022-09-27T15:50:59.5032867Z First line of error 2022-09-27T15:50:59.5033622Z next line of error 2022-09-27T15:50:59.5034002Z last line of error 2022-09-27T15:50:59.5034230Z 2022-09-27T15:50:59.5070847Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:50:59.5072492Z ValueError('\nFirst line of error \n next line of error \n last line of error') 2022-09-27T15:50:59.5073285Z Traceback (most recent call last): 2022-09-27T15:50:59.5074774Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:50:59.5075520Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:50:59.5076545Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func_escape 2022-09-27T15:50:59.5077214Z raise ValueError(expected_err_escape) 2022-09-27T15:50:59.5077644Z ValueError: 2022-09-27T15:50:59.5078016Z First line of error 2022-09-27T15:50:59.5078383Z next line of error 2022-09-27T15:50:59.5078759Z last line of error 2022-09-27T15:50:59.5078996Z 2022-09-27T15:50:59.8981238Z ok (2.455s) 2022-09-27T15:50:59.8981462Z 2022-09-27T15:50:59.8981958Z ---------------------------------------------------------------------- 2022-09-27T15:50:59.8982405Z Ran 1 test in 2.455s 2022-09-27T15:50:59.8982525Z 2022-09-27T15:50:59.8982577Z OK 2022-09-27T15:50:59.8982672Z 2022-09-27T15:50:59.8982766Z Generating XML reports... 2022-09-27T15:50:59.9017751Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155057.xml 2022-09-27T15:51:01.2623943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:01.2624326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:01.2632447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:01.2632811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:01.4380069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbt4f1f_ 2022-09-27T15:51:01.4381068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbt4f1f_/_remote_module_non_scriptable.py 2022-09-27T15:51:01.7038780Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:01.7048637Z 2022-09-27T15:51:01.7048756Z Running tests... 2022-09-27T15:51:01.7049226Z ---------------------------------------------------------------------- 2022-09-27T15:51:01.9480880Z test_py_rpc_rref_args (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30836 2022-09-27T15:51:01.9508773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30837 2022-09-27T15:51:01.9536792Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30838 2022-09-27T15:51:01.9568254Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30839 2022-09-27T15:51:03.0675896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:03.0676583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:03.0707521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:03.0708173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:03.0873346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:03.0873918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:03.0897514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:03.0898039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:03.1061525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:03.1062316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:03.1106186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:03.1106847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:03.1274078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:03.1274906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:03.1295422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:03.1296021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:03.2330302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79y_4lzm 2022-09-27T15:51:03.2331073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79y_4lzm/_remote_module_non_scriptable.py 2022-09-27T15:51:03.2378621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5pxot3h 2022-09-27T15:51:03.2379937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5pxot3h/_remote_module_non_scriptable.py 2022-09-27T15:51:03.2542374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpse95czl0 2022-09-27T15:51:03.2543519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpse95czl0/_remote_module_non_scriptable.py 2022-09-27T15:51:03.2563677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2v7xh7l5 2022-09-27T15:51:03.2565440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2v7xh7l5/_remote_module_non_scriptable.py 2022-09-27T15:51:03.4905777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:51:03.4974754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:03.5120503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:03.5319296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:04.2626667Z ok (2.557s) 2022-09-27T15:51:04.2626923Z 2022-09-27T15:51:04.2627445Z ---------------------------------------------------------------------- 2022-09-27T15:51:04.2627772Z Ran 1 test in 2.558s 2022-09-27T15:51:04.2627887Z 2022-09-27T15:51:04.2627935Z OK 2022-09-27T15:51:04.2628030Z 2022-09-27T15:51:04.2628127Z Generating XML reports... 2022-09-27T15:51:04.2663061Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155101.xml 2022-09-27T15:51:05.6401707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:05.6402342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:05.6411354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:05.6411963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:05.8064654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexr2zehm 2022-09-27T15:51:05.8065353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexr2zehm/_remote_module_non_scriptable.py 2022-09-27T15:51:06.0739823Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:06.0750062Z 2022-09-27T15:51:06.0750448Z Running tests... 2022-09-27T15:51:06.0751047Z ---------------------------------------------------------------------- 2022-09-27T15:51:06.3184643Z test_py_rref_args (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31059 2022-09-27T15:51:06.3211584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31060 2022-09-27T15:51:06.3238922Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31061 2022-09-27T15:51:06.3268414Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31062 2022-09-27T15:51:07.4397576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:07.4398225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:07.4403388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:07.4403983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:07.4783806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:07.4784578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:07.4789473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:07.4790107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:07.4850338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:07.4851006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:07.5220560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:07.5263585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:07.5264523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:07.5265198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:07.5635540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:07.5636185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:07.6059291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg3xu20l0 2022-09-27T15:51:07.6060098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg3xu20l0/_remote_module_non_scriptable.py 2022-09-27T15:51:07.6273626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98b2d5yf 2022-09-27T15:51:07.6274759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98b2d5yf/_remote_module_non_scriptable.py 2022-09-27T15:51:07.6508459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplu6weer8 2022-09-27T15:51:07.6509227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplu6weer8/_remote_module_non_scriptable.py 2022-09-27T15:51:07.6938443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qno3cfk 2022-09-27T15:51:07.6939447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qno3cfk/_remote_module_non_scriptable.py 2022-09-27T15:51:07.8677978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:07.8881432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:51:07.9129352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:07.9577012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:08.6324967Z ok (2.557s) 2022-09-27T15:51:08.6325263Z 2022-09-27T15:51:08.6325757Z ---------------------------------------------------------------------- 2022-09-27T15:51:08.6326035Z Ran 1 test in 2.557s 2022-09-27T15:51:08.6326151Z 2022-09-27T15:51:08.6326471Z OK 2022-09-27T15:51:08.6326725Z 2022-09-27T15:51:08.6326826Z Generating XML reports... 2022-09-27T15:51:08.6362611Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155106.xml 2022-09-27T15:51:10.0088805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:10.0089463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:10.0098064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:10.0098696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:10.1654196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzsjbgazb 2022-09-27T15:51:10.1655135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzsjbgazb/_remote_module_non_scriptable.py 2022-09-27T15:51:10.4282076Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:10.4291895Z 2022-09-27T15:51:10.4292504Z Running tests... 2022-09-27T15:51:10.4293087Z ---------------------------------------------------------------------- 2022-09-27T15:51:10.6727658Z test_py_rref_args_user_share (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31282 2022-09-27T15:51:10.6757010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31283 2022-09-27T15:51:10.6786909Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31284 2022-09-27T15:51:10.6815030Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31285 2022-09-27T15:51:11.7561471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:11.7562135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:11.7686805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:11.7687345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:11.7921430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:11.7922093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:11.8009315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:11.8009924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:11.8055959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:11.8056669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:11.8364382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:11.8364982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:11.8417412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:11.8418046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:11.8792170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:11.8792823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:11.9199318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17mm7hnt 2022-09-27T15:51:11.9200173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17mm7hnt/_remote_module_non_scriptable.py 2022-09-27T15:51:11.9327456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67rjoaca 2022-09-27T15:51:11.9328383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67rjoaca/_remote_module_non_scriptable.py 2022-09-27T15:51:11.9645447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpml4xnenw 2022-09-27T15:51:11.9646286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpml4xnenw/_remote_module_non_scriptable.py 2022-09-27T15:51:12.0074894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6e1p267e 2022-09-27T15:51:12.0075739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6e1p267e/_remote_module_non_scriptable.py 2022-09-27T15:51:12.1794420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:12.1926406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:12.2249423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:51:12.2707976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:12.9874324Z ok (2.558s) 2022-09-27T15:51:12.9874471Z 2022-09-27T15:51:12.9874932Z ---------------------------------------------------------------------- 2022-09-27T15:51:12.9875188Z Ran 1 test in 2.558s 2022-09-27T15:51:12.9875305Z 2022-09-27T15:51:12.9875366Z OK 2022-09-27T15:51:12.9875499Z 2022-09-27T15:51:12.9875597Z Generating XML reports... 2022-09-27T15:51:12.9910442Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155110.xml 2022-09-27T15:51:14.3463275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:14.3463659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:14.3471657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:14.3472034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:14.5084939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0y9jf8dx 2022-09-27T15:51:14.5085838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0y9jf8dx/_remote_module_non_scriptable.py 2022-09-27T15:51:14.7755266Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:14.7764974Z 2022-09-27T15:51:14.7765154Z Running tests... 2022-09-27T15:51:14.7765573Z ---------------------------------------------------------------------- 2022-09-27T15:51:15.0234875Z test_py_tensors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31505 2022-09-27T15:51:15.0263480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31506 2022-09-27T15:51:15.0291874Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31507 2022-09-27T15:51:15.0323084Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31508 2022-09-27T15:51:16.1115359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:16.1115966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:16.1198162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:16.1198858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:16.1203031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:16.1203772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:16.1497622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:16.1498593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:16.1525418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:16.1526222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:16.1582495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:16.1583106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:16.1583891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:16.1584423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:16.1877890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:16.1878550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:16.2758817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxoq7jq2 2022-09-27T15:51:16.2759823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxoq7jq2/_remote_module_non_scriptable.py 2022-09-27T15:51:16.2856151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoecmn76u 2022-09-27T15:51:16.2857018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoecmn76u/_remote_module_non_scriptable.py 2022-09-27T15:51:16.2858482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0j628q08 2022-09-27T15:51:16.2861472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0j628q08/_remote_module_non_scriptable.py 2022-09-27T15:51:16.3135704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm2his1qw 2022-09-27T15:51:16.3136937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm2his1qw/_remote_module_non_scriptable.py 2022-09-27T15:51:16.5345579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:16.5422357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:16.5453042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:16.5712177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:51:17.2379278Z ok (2.461s) 2022-09-27T15:51:17.2379539Z 2022-09-27T15:51:17.2380046Z ---------------------------------------------------------------------- 2022-09-27T15:51:17.2380354Z Ran 1 test in 2.461s 2022-09-27T15:51:17.2380460Z 2022-09-27T15:51:17.2380522Z OK 2022-09-27T15:51:17.2380615Z 2022-09-27T15:51:17.2380709Z Generating XML reports... 2022-09-27T15:51:17.2415783Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155114.xml 2022-09-27T15:51:18.6230391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:18.6230788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:18.6239531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:18.6239889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:18.7847572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphww7_pyr 2022-09-27T15:51:18.7848555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphww7_pyr/_remote_module_non_scriptable.py 2022-09-27T15:51:19.0494812Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:19.0504254Z 2022-09-27T15:51:19.0504386Z Running tests... 2022-09-27T15:51:19.0505018Z ---------------------------------------------------------------------- 2022-09-27T15:51:19.2936242Z test_py_tensors_in_container (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31728 2022-09-27T15:51:19.2963750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31729 2022-09-27T15:51:19.2991147Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31730 2022-09-27T15:51:19.3020721Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31731 2022-09-27T15:51:20.3855817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:20.3856444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:20.4056193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:20.4056836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:20.4154786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:20.4155311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:20.4208225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:20.4208898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:20.4379913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:20.4380437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:20.4432408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:20.4433053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:20.4525318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:20.4525876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:20.4737614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:20.4738233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:20.5470746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizat148e 2022-09-27T15:51:20.5472241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizat148e/_remote_module_non_scriptable.py 2022-09-27T15:51:20.5697803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnbzmf3o 2022-09-27T15:51:20.5698591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnbzmf3o/_remote_module_non_scriptable.py 2022-09-27T15:51:20.5790944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiijxk57b 2022-09-27T15:51:20.5792283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiijxk57b/_remote_module_non_scriptable.py 2022-09-27T15:51:20.6002264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkljby23j 2022-09-27T15:51:20.6003625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkljby23j/_remote_module_non_scriptable.py 2022-09-27T15:51:20.8059956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:20.8306826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:20.8415040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:51:20.8626505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:21.5078897Z ok (2.457s) 2022-09-27T15:51:21.5079159Z 2022-09-27T15:51:21.5079936Z ---------------------------------------------------------------------- 2022-09-27T15:51:21.5080214Z Ran 1 test in 2.457s 2022-09-27T15:51:21.5080320Z 2022-09-27T15:51:21.5080381Z OK 2022-09-27T15:51:21.5080549Z 2022-09-27T15:51:21.5080643Z Generating XML reports... 2022-09-27T15:51:21.5115163Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155119.xml 2022-09-27T15:51:22.9168935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:22.9169306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:22.9177128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:22.9177526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:23.0742709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpncd5a7uj 2022-09-27T15:51:23.0743408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpncd5a7uj/_remote_module_non_scriptable.py 2022-09-27T15:51:23.3340986Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:23.3350218Z 2022-09-27T15:51:23.3350321Z Running tests... 2022-09-27T15:51:23.3350759Z ---------------------------------------------------------------------- 2022-09-27T15:51:23.5791176Z test_py_tensors_multi_async_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31951 2022-09-27T15:51:23.5819748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31952 2022-09-27T15:51:23.5846910Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31953 2022-09-27T15:51:23.5876696Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31954 2022-09-27T15:51:24.7438628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:24.7439382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:24.7460933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:24.7461568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:24.7462342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:24.7462929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:24.7767571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:24.7768182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:24.7816643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:24.7817383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:24.7842226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:24.7842822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:24.7843712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:24.7844285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:24.8131145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:24.8131741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:24.9078928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp802fs_te 2022-09-27T15:51:24.9080479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp802fs_te/_remote_module_non_scriptable.py 2022-09-27T15:51:24.9103904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8icia_8 2022-09-27T15:51:24.9106493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8icia_8/_remote_module_non_scriptable.py 2022-09-27T15:51:24.9114686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspijb69m 2022-09-27T15:51:24.9117314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspijb69m/_remote_module_non_scriptable.py 2022-09-27T15:51:24.9396475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp237awa5q 2022-09-27T15:51:24.9397930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp237awa5q/_remote_module_non_scriptable.py 2022-09-27T15:51:25.1628938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:25.1663536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:25.1828470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:51:25.1992333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:25.8934364Z ok (2.558s) 2022-09-27T15:51:25.8934603Z 2022-09-27T15:51:25.8935109Z ---------------------------------------------------------------------- 2022-09-27T15:51:25.8935487Z Ran 1 test in 2.558s 2022-09-27T15:51:25.8935607Z 2022-09-27T15:51:25.8935668Z OK 2022-09-27T15:51:25.8935760Z 2022-09-27T15:51:25.8935852Z Generating XML reports... 2022-09-27T15:51:25.8971436Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155123.xml 2022-09-27T15:51:27.2628132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:27.2628518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:27.2636009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:27.2636579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:27.4186535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rhiv79d 2022-09-27T15:51:27.4187556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rhiv79d/_remote_module_non_scriptable.py 2022-09-27T15:51:27.6808062Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:27.6817388Z 2022-09-27T15:51:27.6817523Z Running tests... 2022-09-27T15:51:27.6818239Z ---------------------------------------------------------------------- 2022-09-27T15:51:27.9301646Z test_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32174 2022-09-27T15:51:27.9329255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32175 2022-09-27T15:51:27.9357667Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32176 2022-09-27T15:51:27.9387409Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32177 2022-09-27T15:51:29.0527995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:29.0528625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:29.0898467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:29.0899140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:29.1344220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:29.1344901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:29.1350074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:29.1350952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:29.1593328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:29.1593983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:29.1732011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:29.1732553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:29.1733572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:29.1734060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:29.1964087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:29.1964680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:29.2184103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0xfaf4q 2022-09-27T15:51:29.2185296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0xfaf4q/_remote_module_non_scriptable.py 2022-09-27T15:51:29.3004962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qurl2em 2022-09-27T15:51:29.3005893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56d952uk 2022-09-27T15:51:29.3006652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qurl2em/_remote_module_non_scriptable.py 2022-09-27T15:51:29.3007384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56d952uk/_remote_module_non_scriptable.py 2022-09-27T15:51:29.3238794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_teqt9bz 2022-09-27T15:51:29.3239711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_teqt9bz/_remote_module_non_scriptable.py 2022-09-27T15:51:29.4985098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:29.5613846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:51:29.5619395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:29.5841128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:30.2445322Z ok (2.562s) 2022-09-27T15:51:30.2445601Z 2022-09-27T15:51:30.2446059Z ---------------------------------------------------------------------- 2022-09-27T15:51:30.2446311Z Ran 1 test in 2.563s 2022-09-27T15:51:30.2446425Z 2022-09-27T15:51:30.2446484Z OK 2022-09-27T15:51:30.2446574Z 2022-09-27T15:51:30.2446671Z Generating XML reports... 2022-09-27T15:51:30.2481274Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155127.xml 2022-09-27T15:51:31.6380704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:31.6381355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:31.6390955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:31.6391641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:31.8008742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zvo25ma 2022-09-27T15:51:31.8009613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zvo25ma/_remote_module_non_scriptable.py 2022-09-27T15:51:32.0667787Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:32.0678408Z 2022-09-27T15:51:32.0679119Z Running tests... 2022-09-27T15:51:32.0679547Z ---------------------------------------------------------------------- 2022-09-27T15:51:32.3140234Z test_py_user_defined (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32397 2022-09-27T15:51:32.3168006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32398 2022-09-27T15:51:32.3195723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32399 2022-09-27T15:51:32.3226610Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32400 2022-09-27T15:51:33.4047900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:33.4048481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:33.4326880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:33.4327519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:33.4385091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:33.4385719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:33.4476102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:33.4476701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:33.4547600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:33.4548113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:33.4712978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:33.4713613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:33.4766950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:33.4767562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:33.4911513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:33.4912187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:33.5736429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy71yansa 2022-09-27T15:51:33.5737166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy71yansa/_remote_module_non_scriptable.py 2022-09-27T15:51:33.5996334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbj7qc_y1 2022-09-27T15:51:33.5997130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbj7qc_y1/_remote_module_non_scriptable.py 2022-09-27T15:51:33.6029817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6e8xv2zz 2022-09-27T15:51:33.6031156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6e8xv2zz/_remote_module_non_scriptable.py 2022-09-27T15:51:33.6179763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmlwd5s7j 2022-09-27T15:51:33.6180944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmlwd5s7j/_remote_module_non_scriptable.py 2022-09-27T15:51:33.8285350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:33.8564979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:33.8572000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:51:33.8728586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:34.5282917Z ok (2.460s) 2022-09-27T15:51:34.5283193Z 2022-09-27T15:51:34.5283639Z ---------------------------------------------------------------------- 2022-09-27T15:51:34.5283987Z Ran 1 test in 2.460s 2022-09-27T15:51:34.5284102Z 2022-09-27T15:51:34.5284149Z OK 2022-09-27T15:51:34.5284241Z 2022-09-27T15:51:34.5284334Z Generating XML reports... 2022-09-27T15:51:34.5319182Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155132.xml 2022-09-27T15:51:35.9182576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:35.9183207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:35.9191113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:35.9191782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:36.0723550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mghs0kh 2022-09-27T15:51:36.0724231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mghs0kh/_remote_module_non_scriptable.py 2022-09-27T15:51:36.3323932Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:36.3334121Z 2022-09-27T15:51:36.3334405Z Running tests... 2022-09-27T15:51:36.3335033Z ---------------------------------------------------------------------- 2022-09-27T15:51:36.5770922Z test_register_rpc_backend_and_set_and_start_rpc_backend (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32620 2022-09-27T15:51:36.5799414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32621 2022-09-27T15:51:36.5828497Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32622 2022-09-27T15:51:36.5859642Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32623 2022-09-27T15:51:37.7159928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:37.7160539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:37.7530310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:37.7530912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:37.7827391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:37.7828048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:37.7871516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:37.7872144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:37.7966429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:37.7967051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:37.8209219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:37.8209884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:37.8249626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:37.8250291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:37.8344413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:37.8345022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:37.8806266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1k28ro5l 2022-09-27T15:51:37.8807067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1k28ro5l/_remote_module_non_scriptable.py 2022-09-27T15:51:37.9476315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6gv3oof 2022-09-27T15:51:37.9477135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6gv3oof/_remote_module_non_scriptable.py 2022-09-27T15:51:37.9508506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwviuu7r8 2022-09-27T15:51:37.9509848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwviuu7r8/_remote_module_non_scriptable.py 2022-09-27T15:51:37.9609464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qaopw7x 2022-09-27T15:51:37.9610934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qaopw7x/_remote_module_non_scriptable.py 2022-09-27T15:51:38.1356929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:38.2024593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:51:38.2076843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:38.2182953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:38.5910930Z ok (2.257s) 2022-09-27T15:51:38.5911212Z 2022-09-27T15:51:38.5911756Z ---------------------------------------------------------------------- 2022-09-27T15:51:38.5912008Z Ran 1 test in 2.258s 2022-09-27T15:51:38.5912125Z 2022-09-27T15:51:38.5912172Z OK 2022-09-27T15:51:38.5912262Z 2022-09-27T15:51:38.5912356Z Generating XML reports... 2022-09-27T15:51:38.5966911Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155136.xml 2022-09-27T15:51:39.9852824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:39.9861753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:39.9862558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:39.9863182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:40.1431845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3j4kbs9x 2022-09-27T15:51:40.1432523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3j4kbs9x/_remote_module_non_scriptable.py 2022-09-27T15:51:40.4068336Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:40.4078498Z 2022-09-27T15:51:40.4078853Z Running tests... 2022-09-27T15:51:40.4079534Z ---------------------------------------------------------------------- 2022-09-27T15:51:40.6545135Z test_reinit (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32679 2022-09-27T15:51:40.6571981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32680 2022-09-27T15:51:40.6601933Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32681 2022-09-27T15:51:40.6631148Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32682 2022-09-27T15:51:41.7632749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:41.7633357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:41.7923891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:41.7924563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:41.7947977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:41.7948973Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:41.8010782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:41.8011677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:41.8097175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:41.8097785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:41.8304280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:41.8304923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:41.8325026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:41.8325669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:41.8458104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:41.8458751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:41.9284988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmlcwvd6 2022-09-27T15:51:41.9286190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmlcwvd6/_remote_module_non_scriptable.py 2022-09-27T15:51:41.9561139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92fle0_r 2022-09-27T15:51:41.9562932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92fle0_r/_remote_module_non_scriptable.py 2022-09-27T15:51:41.9580011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4yuvk7z 2022-09-27T15:51:41.9582017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4yuvk7z/_remote_module_non_scriptable.py 2022-09-27T15:51:41.9720546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo92un_4f 2022-09-27T15:51:41.9721357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo92un_4f/_remote_module_non_scriptable.py 2022-09-27T15:51:42.1854303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:51:42.2109998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:42.2120252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:42.2302123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:42.4830085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:51:42.4832457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:51:42.4833258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:51:42.4834064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:51:42.4835502Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:51:42.4930042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:51:42.4931203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:51:42.4932360Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:51:42.9688844Z ok (2.561s) 2022-09-27T15:51:42.9689099Z 2022-09-27T15:51:42.9689615Z ---------------------------------------------------------------------- 2022-09-27T15:51:42.9690107Z Ran 1 test in 2.561s 2022-09-27T15:51:42.9690228Z 2022-09-27T15:51:42.9690289Z OK 2022-09-27T15:51:42.9690448Z 2022-09-27T15:51:42.9690531Z Generating XML reports... 2022-09-27T15:51:42.9725810Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155140.xml 2022-09-27T15:51:44.3248971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:44.3249332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:44.3258114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:44.3258601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:44.4872826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbznn8qua 2022-09-27T15:51:44.4873895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbznn8qua/_remote_module_non_scriptable.py 2022-09-27T15:51:44.7532227Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:44.7541753Z 2022-09-27T15:51:44.7541887Z Running tests... 2022-09-27T15:51:44.7542881Z ---------------------------------------------------------------------- 2022-09-27T15:51:45.0008558Z test_remote_same_worker (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 446 2022-09-27T15:51:45.0036570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 447 2022-09-27T15:51:45.0063863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 448 2022-09-27T15:51:45.0093940Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 449 2022-09-27T15:51:46.1951522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:46.1952190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:46.2152592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:46.2153315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:46.2321877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:46.2322439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:46.2532217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:46.2532885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:46.2878386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:46.2879096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:46.3233329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:46.3233939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:46.3588771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgb0uw08z 2022-09-27T15:51:46.3589935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgb0uw08z/_remote_module_non_scriptable.py 2022-09-27T15:51:46.3618729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:46.3619361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:46.3814136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpur4b1rup 2022-09-27T15:51:46.3814814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpur4b1rup/_remote_module_non_scriptable.py 2022-09-27T15:51:46.4010911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:46.4011741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:46.4515437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa97sxg4a 2022-09-27T15:51:46.4516225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa97sxg4a/_remote_module_non_scriptable.py 2022-09-27T15:51:46.5294102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6zpfkzy 2022-09-27T15:51:46.5295019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6zpfkzy/_remote_module_non_scriptable.py 2022-09-27T15:51:46.6212623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:51:46.6404217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:46.7094029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:46.7889384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:47.4152531Z ok (2.661s) 2022-09-27T15:51:47.4152773Z 2022-09-27T15:51:47.4153237Z ---------------------------------------------------------------------- 2022-09-27T15:51:47.4153621Z Ran 1 test in 2.661s 2022-09-27T15:51:47.4153799Z 2022-09-27T15:51:47.4153892Z OK 2022-09-27T15:51:47.4154042Z 2022-09-27T15:51:47.4154192Z Generating XML reports... 2022-09-27T15:51:47.4189018Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155144.xml 2022-09-27T15:51:48.7849443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:48.7849822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:48.7858091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:48.7858615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:48.9446099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6nory00j 2022-09-27T15:51:48.9446762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6nory00j/_remote_module_non_scriptable.py 2022-09-27T15:51:49.2088070Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:49.2097713Z 2022-09-27T15:51:49.2097992Z Running tests... 2022-09-27T15:51:49.2098441Z ---------------------------------------------------------------------- 2022-09-27T15:51:49.4575869Z test_remote_throw (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 670 2022-09-27T15:51:49.4603786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 671 2022-09-27T15:51:49.4631621Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 672 2022-09-27T15:51:49.4661654Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 673 2022-09-27T15:51:50.5717078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:50.5717718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:50.5808239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:50.5808763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:50.6105861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:50.6106595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:50.6203395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:50.6204057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:50.6463654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:50.6464368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:50.6568730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:50.6569240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:50.6858483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:50.6859113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:50.6953524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:50.6954138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:50.7385479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphp5e8w4f 2022-09-27T15:51:50.7386226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphp5e8w4f/_remote_module_non_scriptable.py 2022-09-27T15:51:50.7502040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6b57fu8l 2022-09-27T15:51:50.7502761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6b57fu8l/_remote_module_non_scriptable.py 2022-09-27T15:51:50.8125657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwvrprsrs 2022-09-27T15:51:50.8126777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwvrprsrs/_remote_module_non_scriptable.py 2022-09-27T15:51:50.8226791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf52qu9mf 2022-09-27T15:51:50.8227886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf52qu9mf/_remote_module_non_scriptable.py 2022-09-27T15:51:50.9969144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:51:51.0106092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:51.0713673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:51.0981169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:51.3072776Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:51:51.3073512Z ValueError('Expected error') 2022-09-27T15:51:51.3073990Z Traceback (most recent call last): 2022-09-27T15:51:51.3075097Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:51:51.3078392Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:51:51.3080192Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc 2022-09-27T15:51:51.3080927Z raise ValueError("Expected error") 2022-09-27T15:51:51.3081498Z ValueError: Expected error 2022-09-27T15:51:51.3081904Z 2022-09-27T15:51:51.3226713Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:51:51.3227414Z ValueError('Expected error') 2022-09-27T15:51:51.3227897Z Traceback (most recent call last): 2022-09-27T15:51:51.3228776Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:51:51.3229501Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:51:51.3230464Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc 2022-09-27T15:51:51.3231127Z raise ValueError("Expected error") 2022-09-27T15:51:51.3231633Z ValueError: Expected error 2022-09-27T15:51:51.3231891Z 2022-09-27T15:51:51.3270428Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:51:51.3271002Z ValueError('Expected error') 2022-09-27T15:51:51.3271456Z Traceback (most recent call last): 2022-09-27T15:51:51.3272330Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:51:51.3272981Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:51:51.3273943Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc 2022-09-27T15:51:51.3274785Z raise ValueError("Expected error") 2022-09-27T15:51:51.3275251Z ValueError: Expected error 2022-09-27T15:51:51.3275519Z 2022-09-27T15:51:51.3347890Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:51:51.3348540Z ValueError('Expected error') 2022-09-27T15:51:51.3349015Z Traceback (most recent call last): 2022-09-27T15:51:51.3349902Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:51:51.3350610Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:51:51.3351659Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc 2022-09-27T15:51:51.3352339Z raise ValueError("Expected error") 2022-09-27T15:51:51.3352796Z ValueError: Expected error 2022-09-27T15:51:51.3353033Z 2022-09-27T15:51:51.7721242Z ok (2.562s) 2022-09-27T15:51:51.7721476Z 2022-09-27T15:51:51.7722001Z ---------------------------------------------------------------------- 2022-09-27T15:51:51.7722405Z Ran 1 test in 2.562s 2022-09-27T15:51:51.7722529Z 2022-09-27T15:51:51.7722591Z OK 2022-09-27T15:51:51.7722685Z 2022-09-27T15:51:51.7722782Z Generating XML reports... 2022-09-27T15:51:51.7757622Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155149.xml 2022-09-27T15:51:53.1512906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:53.1513325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:53.1521489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:53.1521850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:53.3067201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qkers35 2022-09-27T15:51:53.3068235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qkers35/_remote_module_non_scriptable.py 2022-09-27T15:51:53.5674892Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:53.5684575Z 2022-09-27T15:51:53.5684699Z Running tests... 2022-09-27T15:51:53.5685110Z ---------------------------------------------------------------------- 2022-09-27T15:51:53.8053710Z test_remote_with_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 893 2022-09-27T15:51:53.8081063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 894 2022-09-27T15:51:53.8108812Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 895 2022-09-27T15:51:53.8139178Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 896 2022-09-27T15:51:54.9365663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:54.9366329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:54.9372316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:54.9372946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:54.9736809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:54.9737525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:54.9738526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:54.9739206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:54.9745921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:54.9746487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:54.9747236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:54.9747830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:55.0107570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:55.0108333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:55.0113562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:55.0114164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:55.1010356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnk4fq_2q 2022-09-27T15:51:55.1011106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnk4fq_2q/_remote_module_non_scriptable.py 2022-09-27T15:51:55.1136297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpad9tp2bm 2022-09-27T15:51:55.1137533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpad9tp2bm/_remote_module_non_scriptable.py 2022-09-27T15:51:55.1379113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzl3777qk 2022-09-27T15:51:55.1380213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzl3777qk/_remote_module_non_scriptable.py 2022-09-27T15:51:55.1387402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjonkmnou 2022-09-27T15:51:55.1389697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjonkmnou/_remote_module_non_scriptable.py 2022-09-27T15:51:55.3568067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:55.3745408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:55.3918837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:55.3957540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:51:55.6065248Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:51:55.6066231Z ValueError('Expected error') 2022-09-27T15:51:55.6066711Z Traceback (most recent call last): 2022-09-27T15:51:55.6068332Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:51:55.6069061Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:51:55.6070029Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:51:55.6070697Z raise ValueError(expected_err) 2022-09-27T15:51:55.6071146Z ValueError: Expected error 2022-09-27T15:51:55.6071467Z 2022-09-27T15:51:55.6192775Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:51:55.6193662Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-09-27T15:51:55.6194212Z Traceback (most recent call last): 2022-09-27T15:51:55.6195282Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:51:55.6195985Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:51:55.6196971Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-09-27T15:51:55.6197325Z 2022-09-27T15:51:55.6260611Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:51:55.6261328Z ValueError('Expected error') 2022-09-27T15:51:55.6263213Z Traceback (most recent call last): 2022-09-27T15:51:55.6264505Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:51:55.6265276Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:51:55.6266246Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:51:55.6266914Z raise ValueError(expected_err) 2022-09-27T15:51:55.6267366Z ValueError: Expected error 2022-09-27T15:51:55.6267629Z 2022-09-27T15:51:55.6374931Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:51:55.6375671Z ValueError('Expected error') 2022-09-27T15:51:55.6376139Z Traceback (most recent call last): 2022-09-27T15:51:55.6377073Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:51:55.6377781Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:51:55.6378773Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:51:55.6379438Z raise ValueError(expected_err) 2022-09-27T15:51:55.6379864Z ValueError: Expected error 2022-09-27T15:51:55.6380134Z 2022-09-27T15:51:55.6389724Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:51:55.6390392Z ValueError('Expected error') 2022-09-27T15:51:55.6390830Z Traceback (most recent call last): 2022-09-27T15:51:55.6391753Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:51:55.6392480Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:51:55.6393461Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:51:55.6394093Z raise ValueError(expected_err) 2022-09-27T15:51:55.6394750Z ValueError: Expected error 2022-09-27T15:51:55.6395019Z 2022-09-27T15:51:55.6680386Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:51:55.6681117Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-09-27T15:51:55.6681495Z Traceback (most recent call last): 2022-09-27T15:51:55.6681937Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:51:55.6682274Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:51:55.6682613Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-09-27T15:51:55.6682774Z 2022-09-27T15:51:55.6761829Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:51:55.6763174Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-09-27T15:51:55.6763591Z Traceback (most recent call last): 2022-09-27T15:51:55.6764145Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:51:55.6764626Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:51:55.6764918Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-09-27T15:51:55.6765079Z 2022-09-27T15:51:55.6827928Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:51:55.6829024Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-09-27T15:51:55.6829623Z Traceback (most recent call last): 2022-09-27T15:51:55.6830343Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:51:55.6831063Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:51:55.6831726Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-09-27T15:51:55.6832086Z 2022-09-27T15:51:56.1197549Z ok (2.551s) 2022-09-27T15:51:56.1197809Z 2022-09-27T15:51:56.1198529Z ---------------------------------------------------------------------- 2022-09-27T15:51:56.1198926Z Ran 1 test in 2.551s 2022-09-27T15:51:56.1199189Z 2022-09-27T15:51:56.1199292Z OK 2022-09-27T15:51:56.1199447Z 2022-09-27T15:51:56.1199596Z Generating XML reports... 2022-09-27T15:51:56.1235826Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155153.xml 2022-09-27T15:51:57.4703943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:57.4704612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:57.4713263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:57.4713645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:57.6284761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbp3pjioj 2022-09-27T15:51:57.6286034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbp3pjioj/_remote_module_non_scriptable.py 2022-09-27T15:51:57.8904907Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:51:57.8914822Z 2022-09-27T15:51:57.8915016Z Running tests... 2022-09-27T15:51:57.8915517Z ---------------------------------------------------------------------- 2022-09-27T15:51:58.1323887Z test_return_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1116 2022-09-27T15:51:58.1351569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1117 2022-09-27T15:51:58.1380041Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1118 2022-09-27T15:51:58.1408792Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1119 2022-09-27T15:51:59.2972657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:59.2973307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:59.3158405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:59.3159049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:59.3165875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:59.3166511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:59.3167289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:51:59.3167896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:51:59.3340319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:59.3340859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:59.3542963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:59.3543544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:59.3544286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:59.3544877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:59.3546459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:51:59.3547054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:51:59.4624258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm2k_erfc 2022-09-27T15:51:59.4625169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm2k_erfc/_remote_module_non_scriptable.py 2022-09-27T15:51:59.4808693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xhwxmoo 2022-09-27T15:51:59.4809534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn85jlogi 2022-09-27T15:51:59.4810109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphnzh06uz 2022-09-27T15:51:59.4810699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xhwxmoo/_remote_module_non_scriptable.py 2022-09-27T15:51:59.4811357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn85jlogi/_remote_module_non_scriptable.py 2022-09-27T15:51:59.4812039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphnzh06uz/_remote_module_non_scriptable.py 2022-09-27T15:51:59.7203314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:51:59.7360186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:51:59.7360711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:51:59.7366287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:52:00.3464546Z ok (2.455s) 2022-09-27T15:52:00.3464807Z 2022-09-27T15:52:00.3465291Z ---------------------------------------------------------------------- 2022-09-27T15:52:00.3465667Z Ran 1 test in 2.455s 2022-09-27T15:52:00.3465785Z 2022-09-27T15:52:00.3465847Z OK 2022-09-27T15:52:00.3465940Z 2022-09-27T15:52:00.3466036Z Generating XML reports... 2022-09-27T15:52:00.3501056Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155157.xml 2022-09-27T15:52:01.7270536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:01.7271173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:01.7280991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:01.7281641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:01.8892634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6rsz062 2022-09-27T15:52:01.8893572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6rsz062/_remote_module_non_scriptable.py 2022-09-27T15:52:02.1543513Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:52:02.1553171Z 2022-09-27T15:52:02.1553590Z Running tests... 2022-09-27T15:52:02.1553996Z ---------------------------------------------------------------------- 2022-09-27T15:52:02.4029979Z test_return_future_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1339 2022-09-27T15:52:02.4058009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1340 2022-09-27T15:52:02.4085772Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1341 2022-09-27T15:52:02.4114463Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1342 2022-09-27T15:52:03.4601831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:03.4602481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:03.4959902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:03.4960526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:03.5065106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:03.5066018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:03.5340818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:03.5341725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:03.5414984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:03.5415658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:03.5420753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:03.5421431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:03.5705434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:03.5706106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:03.5782352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:03.5783029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:03.6240896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw76eelgq 2022-09-27T15:52:03.6242061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw76eelgq/_remote_module_non_scriptable.py 2022-09-27T15:52:03.6673881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54vep2gq 2022-09-27T15:52:03.6674856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54vep2gq/_remote_module_non_scriptable.py 2022-09-27T15:52:03.6971599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsiiunrb 2022-09-27T15:52:03.6972647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsiiunrb/_remote_module_non_scriptable.py 2022-09-27T15:52:03.7047793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7qrx2hl 2022-09-27T15:52:03.7049143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7qrx2hl/_remote_module_non_scriptable.py 2022-09-27T15:52:03.8845220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:52:03.9264088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:52:03.9549753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:52:03.9656486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:52:04.6171280Z ok (2.461s) 2022-09-27T15:52:04.6171568Z 2022-09-27T15:52:04.6171944Z ---------------------------------------------------------------------- 2022-09-27T15:52:04.6172195Z Ran 1 test in 2.462s 2022-09-27T15:52:04.6172312Z 2022-09-27T15:52:04.6179859Z OK 2022-09-27T15:52:04.6180031Z 2022-09-27T15:52:04.6180136Z Generating XML reports... 2022-09-27T15:52:04.6207951Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155202.xml 2022-09-27T15:52:05.9755601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:05.9755983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:05.9763691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:05.9764226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:06.1302680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphvdu1ujy 2022-09-27T15:52:06.1303858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphvdu1ujy/_remote_module_non_scriptable.py 2022-09-27T15:52:06.3885718Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:52:06.3895083Z 2022-09-27T15:52:06.3895184Z Running tests... 2022-09-27T15:52:06.3896184Z ---------------------------------------------------------------------- 2022-09-27T15:52:06.6322220Z test_return_future_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1562 2022-09-27T15:52:06.6350126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1563 2022-09-27T15:52:06.6378468Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1564 2022-09-27T15:52:06.6408376Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1565 2022-09-27T15:52:07.8117555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:07.8118257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:07.8500101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:07.8500794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:07.8792182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:07.8792861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:07.9030948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:07.9031520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:07.9160918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:07.9161626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:07.9172405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:07.9173010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:07.9418659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:07.9419337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:07.9547979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:07.9548682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:07.9797744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaadp4117 2022-09-27T15:52:07.9798530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaadp4117/_remote_module_non_scriptable.py 2022-09-27T15:52:08.0441978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp841sbrvc 2022-09-27T15:52:08.0443286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp841sbrvc/_remote_module_non_scriptable.py 2022-09-27T15:52:08.0693989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqr4lec8d 2022-09-27T15:52:08.0695309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqr4lec8d/_remote_module_non_scriptable.py 2022-09-27T15:52:08.0818365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8er7qj1q 2022-09-27T15:52:08.0819106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8er7qj1q/_remote_module_non_scriptable.py 2022-09-27T15:52:08.2405687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:52:08.3007782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:52:08.3319750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:52:08.3384346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:52:09.0468263Z ok (2.657s) 2022-09-27T15:52:09.0468468Z 2022-09-27T15:52:09.0468829Z ---------------------------------------------------------------------- 2022-09-27T15:52:09.0469360Z Ran 1 test in 2.657s 2022-09-27T15:52:09.0469464Z 2022-09-27T15:52:09.0469526Z OK 2022-09-27T15:52:09.0469668Z 2022-09-27T15:52:09.0469771Z Generating XML reports... 2022-09-27T15:52:09.0504619Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155206.xml 2022-09-27T15:52:10.4380983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:10.4381610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:10.4390727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:10.4391384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:10.5987852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_58j2wv 2022-09-27T15:52:10.5988750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_58j2wv/_remote_module_non_scriptable.py 2022-09-27T15:52:10.8640772Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:52:10.8651054Z 2022-09-27T15:52:10.8651496Z Running tests... 2022-09-27T15:52:10.8651903Z ---------------------------------------------------------------------- 2022-09-27T15:52:11.1115432Z test_return_local_rrefs (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1785 2022-09-27T15:52:11.1143167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1786 2022-09-27T15:52:11.1171974Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1787 2022-09-27T15:52:11.1202466Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1788 2022-09-27T15:52:12.2241295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:12.2242007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:12.2265477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:12.2266158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:12.2520256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:12.2520894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:12.2567776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:12.2568491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:12.2637998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:12.2638542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:12.2670184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:12.2670726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:12.2909717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:12.2910281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:12.2949744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:12.2950449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:12.3903048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufbxq653 2022-09-27T15:52:12.3903855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufbxq653/_remote_module_non_scriptable.py 2022-09-27T15:52:12.3946365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxa316ozh 2022-09-27T15:52:12.3948817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxa316ozh/_remote_module_non_scriptable.py 2022-09-27T15:52:12.4181627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_vyc6r0 2022-09-27T15:52:12.4182407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_vyc6r0/_remote_module_non_scriptable.py 2022-09-27T15:52:12.4226316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2hsypx7 2022-09-27T15:52:12.4227786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2hsypx7/_remote_module_non_scriptable.py 2022-09-27T15:52:12.6502229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:52:12.6571538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:52:12.6797537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:52:12.6842929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:52:13.3257757Z ok (2.460s) 2022-09-27T15:52:13.3258005Z 2022-09-27T15:52:13.3258512Z ---------------------------------------------------------------------- 2022-09-27T15:52:13.3258766Z Ran 1 test in 2.461s 2022-09-27T15:52:13.3258882Z 2022-09-27T15:52:13.3258943Z OK 2022-09-27T15:52:13.3259035Z 2022-09-27T15:52:13.3259118Z Generating XML reports... 2022-09-27T15:52:13.3295211Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155210.xml 2022-09-27T15:52:14.7205194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:14.7205625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:14.7213261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:14.7213624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:14.8805355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dpb3ah7 2022-09-27T15:52:14.8806766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dpb3ah7/_remote_module_non_scriptable.py 2022-09-27T15:52:15.1413581Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:52:15.1423396Z 2022-09-27T15:52:15.1423503Z Running tests... 2022-09-27T15:52:15.1424149Z ---------------------------------------------------------------------- 2022-09-27T15:52:15.3845726Z test_rpc_barrier_all (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2008 2022-09-27T15:52:15.3872819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2009 2022-09-27T15:52:15.3900704Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2010 2022-09-27T15:52:15.3930976Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2011 2022-09-27T15:52:16.4836459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:16.4837147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:16.4919452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:16.4920102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:16.5098135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:16.5098817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:16.5105436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:16.5106317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:16.5213621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:16.5214157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:16.5296466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:16.5297002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:16.5483737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:16.5484393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:16.5497581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:16.5498247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:16.6482877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1aoq_4s 2022-09-27T15:52:16.6483526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1aoq_4s/_remote_module_non_scriptable.py 2022-09-27T15:52:16.6552345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznpyn189 2022-09-27T15:52:16.6553816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznpyn189/_remote_module_non_scriptable.py 2022-09-27T15:52:16.6754130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpunuvz5tw 2022-09-27T15:52:16.6755208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp915getx 2022-09-27T15:52:16.6756076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpunuvz5tw/_remote_module_non_scriptable.py 2022-09-27T15:52:16.6756857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp915getx/_remote_module_non_scriptable.py 2022-09-27T15:52:16.9103211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:52:16.9342385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:52:16.9361768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:52:16.9387468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:52:17.5986721Z ok (2.456s) 2022-09-27T15:52:17.5986949Z 2022-09-27T15:52:17.5987377Z ---------------------------------------------------------------------- 2022-09-27T15:52:17.5987755Z Ran 1 test in 2.456s 2022-09-27T15:52:17.5987940Z 2022-09-27T15:52:17.5988019Z OK 2022-09-27T15:52:17.5988287Z 2022-09-27T15:52:17.5988373Z Generating XML reports... 2022-09-27T15:52:17.6023724Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155215.xml 2022-09-27T15:52:18.9747626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:18.9748021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:18.9756391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:18.9756745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:19.1345413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphd2d5yry 2022-09-27T15:52:19.1346718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphd2d5yry/_remote_module_non_scriptable.py 2022-09-27T15:52:19.4016529Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:52:19.4026447Z 2022-09-27T15:52:19.4026762Z Running tests... 2022-09-27T15:52:19.4027351Z ---------------------------------------------------------------------- 2022-09-27T15:52:19.6528599Z test_rpc_barrier_multithreaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2231 2022-09-27T15:52:19.6556845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2232 2022-09-27T15:52:19.6585164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2233 2022-09-27T15:52:19.6615051Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2234 2022-09-27T15:52:20.7521009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:20.7522038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:20.7671173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:20.7671788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:20.7890336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:20.7890943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:20.8029244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:20.8029932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:20.8135862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:20.8136512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:20.8348284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:20.8348917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:20.8496619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:20.8497333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:20.8705610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:20.8706268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:20.9153706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2xhx0749 2022-09-27T15:52:20.9154804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2xhx0749/_remote_module_non_scriptable.py 2022-09-27T15:52:20.9303647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8k82e5w 2022-09-27T15:52:20.9304586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8k82e5w/_remote_module_non_scriptable.py 2022-09-27T15:52:20.9766305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9wvvy1o 2022-09-27T15:52:20.9767281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9wvvy1o/_remote_module_non_scriptable.py 2022-09-27T15:52:20.9967728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsmf5ykoj 2022-09-27T15:52:20.9968927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsmf5ykoj/_remote_module_non_scriptable.py 2022-09-27T15:52:21.1709984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:52:21.2052094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:52:21.2335084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:52:21.2530753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:52:21.8671959Z ok (2.464s) 2022-09-27T15:52:21.8672201Z 2022-09-27T15:52:21.8672665Z ---------------------------------------------------------------------- 2022-09-27T15:52:21.8673045Z Ran 1 test in 2.464s 2022-09-27T15:52:21.8673232Z 2022-09-27T15:52:21.8673322Z OK 2022-09-27T15:52:21.8673469Z 2022-09-27T15:52:21.8673612Z Generating XML reports... 2022-09-27T15:52:21.8709188Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155219.xml 2022-09-27T15:52:23.2677308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:23.2677950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:23.2686456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:23.2687089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:23.4281574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk81i4b2m 2022-09-27T15:52:23.4282445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk81i4b2m/_remote_module_non_scriptable.py 2022-09-27T15:52:23.6963777Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:52:23.6973852Z 2022-09-27T15:52:23.6974345Z Running tests... 2022-09-27T15:52:23.6974753Z ---------------------------------------------------------------------- 2022-09-27T15:52:23.9433040Z test_rpc_barrier_partial_subset (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2466 2022-09-27T15:52:23.9460060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2467 2022-09-27T15:52:23.9487307Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2468 2022-09-27T15:52:23.9516076Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2469 2022-09-27T15:52:25.0939340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:25.0940009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:25.1241317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:25.1241960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:25.1314829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:25.1315535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:25.1597745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:25.1598375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:25.1632267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:25.1632865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:25.1669833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:25.1670438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:25.1998732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:25.1999482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:25.2035429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:25.2036306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:25.2580266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbw92d_w4 2022-09-27T15:52:25.2581369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbw92d_w4/_remote_module_non_scriptable.py 2022-09-27T15:52:25.2857173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1s9hbzi6 2022-09-27T15:52:25.2857920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1s9hbzi6/_remote_module_non_scriptable.py 2022-09-27T15:52:25.3264705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc61k4mbw 2022-09-27T15:52:25.3266137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc61k4mbw/_remote_module_non_scriptable.py 2022-09-27T15:52:25.3293690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpveu0akh0 2022-09-27T15:52:25.3295893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpveu0akh0/_remote_module_non_scriptable.py 2022-09-27T15:52:25.5147463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:52:25.5413454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:52:25.5834322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:52:25.5850457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:52:26.2574022Z ok (2.560s) 2022-09-27T15:52:26.2574221Z 2022-09-27T15:52:26.2574617Z ---------------------------------------------------------------------- 2022-09-27T15:52:26.2574873Z Ran 1 test in 2.560s 2022-09-27T15:52:26.2574991Z 2022-09-27T15:52:26.2575065Z OK 2022-09-27T15:52:26.2575210Z 2022-09-27T15:52:26.2576597Z Generating XML reports... 2022-09-27T15:52:26.2611149Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155223.xml 2022-09-27T15:52:27.6281878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:27.6282269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:27.6290444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:27.6290807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:27.7863769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp843dwp6v 2022-09-27T15:52:27.7865233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp843dwp6v/_remote_module_non_scriptable.py 2022-09-27T15:52:28.0446323Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:52:28.0456187Z 2022-09-27T15:52:28.0456289Z Running tests... 2022-09-27T15:52:28.0457330Z ---------------------------------------------------------------------- 2022-09-27T15:52:28.2914165Z test_rpc_barrier_subset (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2689 2022-09-27T15:52:28.2942524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2690 2022-09-27T15:52:28.2970125Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2691 2022-09-27T15:52:28.2999663Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2692 2022-09-27T15:52:29.3806730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:29.3807347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:29.3838708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:29.3839326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:29.3851006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:29.3851715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:29.4132128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:29.4132824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:29.4192070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:29.4192595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:29.4221136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:29.4221657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:29.4232386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:29.4232882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:29.4517438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:29.4518057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:29.5457539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvrv72nv 2022-09-27T15:52:29.5458321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvrv72nv/_remote_module_non_scriptable.py 2022-09-27T15:52:29.5480388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2e3zo59 2022-09-27T15:52:29.5483093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2e3zo59/_remote_module_non_scriptable.py 2022-09-27T15:52:29.5495864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3let8sqe 2022-09-27T15:52:29.5497719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3let8sqe/_remote_module_non_scriptable.py 2022-09-27T15:52:29.5781979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5e08tjle 2022-09-27T15:52:29.5782711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5e08tjle/_remote_module_non_scriptable.py 2022-09-27T15:52:29.8095815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:52:29.8099649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:52:29.8120466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:52:29.8440078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:52:30.6056893Z ok (2.560s) 2022-09-27T15:52:30.6057177Z 2022-09-27T15:52:30.6057707Z ---------------------------------------------------------------------- 2022-09-27T15:52:30.6058028Z Ran 1 test in 2.560s 2022-09-27T15:52:30.6058145Z 2022-09-27T15:52:30.6058218Z OK 2022-09-27T15:52:30.6058311Z 2022-09-27T15:52:30.6058394Z Generating XML reports... 2022-09-27T15:52:30.6093990Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155228.xml 2022-09-27T15:52:31.9979397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:31.9979779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:31.9987677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:31.9988106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:32.1548801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3hno9rfd 2022-09-27T15:52:32.1549851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3hno9rfd/_remote_module_non_scriptable.py 2022-09-27T15:52:32.4159456Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:52:32.4169528Z 2022-09-27T15:52:32.4169870Z Running tests... 2022-09-27T15:52:32.4170466Z ---------------------------------------------------------------------- 2022-09-27T15:52:32.6609430Z test_rpc_profiling_async_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2912 2022-09-27T15:52:32.6637070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2913 2022-09-27T15:52:32.6665017Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2914 2022-09-27T15:52:32.6695101Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2915 2022-09-27T15:52:33.7971858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:33.7972492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:33.8233235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:33.8233929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:33.8339368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:33.8339981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:33.8611410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:33.8612096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:33.8986923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:33.8987680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:33.8988423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:33.8989008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:33.9364887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:33.9365500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:33.9366284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:33.9366863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:33.9614671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9cxcm5n2 2022-09-27T15:52:33.9615653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9cxcm5n2/_remote_module_non_scriptable.py 2022-09-27T15:52:33.9885226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrdbdrr1 2022-09-27T15:52:33.9886060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrdbdrr1/_remote_module_non_scriptable.py 2022-09-27T15:52:34.0619178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcaoihq2d 2022-09-27T15:52:34.0619941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcaoihq2d/_remote_module_non_scriptable.py 2022-09-27T15:52:34.0623253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42n25exv 2022-09-27T15:52:34.0625583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42n25exv/_remote_module_non_scriptable.py 2022-09-27T15:52:34.2182923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:52:34.2430672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:52:34.3164324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:52:34.3220742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:52:34.5945214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:52:34.6045796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:52:34.6148113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:52:34.6150050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:52:34.6151301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:52:34.6152465Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:52:34.6153999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:52:34.6155304Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:52:36.0771005Z ok (3.660s) 2022-09-27T15:52:36.0771197Z 2022-09-27T15:52:36.0771485Z ---------------------------------------------------------------------- 2022-09-27T15:52:36.0771742Z Ran 1 test in 3.660s 2022-09-27T15:52:36.0771862Z 2022-09-27T15:52:36.0771924Z OK 2022-09-27T15:52:36.0772017Z 2022-09-27T15:52:36.0772113Z Generating XML reports... 2022-09-27T15:52:36.0807801Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155232.xml 2022-09-27T15:52:37.4704244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:37.4704639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:37.4712637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:37.4713005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:37.6261630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qndne8b 2022-09-27T15:52:37.6262524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qndne8b/_remote_module_non_scriptable.py 2022-09-27T15:52:37.8864347Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:52:37.8874462Z 2022-09-27T15:52:37.8874758Z Running tests... 2022-09-27T15:52:37.8875671Z ---------------------------------------------------------------------- 2022-09-27T15:52:38.1307815Z test_rpc_profiling_async_function_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3147 2022-09-27T15:52:38.1335175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3148 2022-09-27T15:52:38.1362134Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3149 2022-09-27T15:52:38.1393098Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3150 2022-09-27T15:52:39.2055623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:39.2056255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:39.2214144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:39.2214830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:39.2432358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:39.2433056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:39.2591525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:39.2592544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:39.2593336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:39.2593952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:39.2594882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:39.2595531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:39.2958831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:39.2959559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:39.2963068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:39.2963626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:39.3704973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhiawkih 2022-09-27T15:52:39.3705903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhiawkih/_remote_module_non_scriptable.py 2022-09-27T15:52:39.3854560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9njolu1 2022-09-27T15:52:39.3855346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9njolu1/_remote_module_non_scriptable.py 2022-09-27T15:52:39.4245958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hz_nl2e 2022-09-27T15:52:39.4246751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8xsdqbn 2022-09-27T15:52:39.4247474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hz_nl2e/_remote_module_non_scriptable.py 2022-09-27T15:52:39.4248182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8xsdqbn/_remote_module_non_scriptable.py 2022-09-27T15:52:39.6277573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:52:39.6426747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:52:39.6818616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:52:39.6839988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:52:39.9185832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:52:39.9287587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:52:39.9293368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:52:39.9297002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:52:39.9298254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:52:39.9391376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:52:39.9392573Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:52:39.9397219Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:52:41.3465377Z ok (3.459s) 2022-09-27T15:52:41.3465581Z 2022-09-27T15:52:41.3466210Z ---------------------------------------------------------------------- 2022-09-27T15:52:41.3466479Z Ran 1 test in 3.459s 2022-09-27T15:52:41.3466598Z 2022-09-27T15:52:41.3466658Z OK 2022-09-27T15:52:41.3466886Z 2022-09-27T15:52:41.3466971Z Generating XML reports... 2022-09-27T15:52:41.3507654Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155237.xml 2022-09-27T15:52:42.7667045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:42.7667419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:42.7675268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:42.7675631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:42.9248283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ekvhgme 2022-09-27T15:52:42.9249023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ekvhgme/_remote_module_non_scriptable.py 2022-09-27T15:52:43.1869350Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:52:43.1879755Z 2022-09-27T15:52:43.1880033Z Running tests... 2022-09-27T15:52:43.1880468Z ---------------------------------------------------------------------- 2022-09-27T15:52:43.4316409Z test_rpc_profiling_remote_record_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3382 2022-09-27T15:52:43.4344238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3383 2022-09-27T15:52:43.4372599Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3384 2022-09-27T15:52:43.4402790Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3385 2022-09-27T15:52:44.5775165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:44.5775869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:44.5999841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:44.6000520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:44.6021632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:44.6022283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:44.6128272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:44.6128799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:44.6276197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:44.6276894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:44.6365835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:44.6366392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:44.6390884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:44.6391497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:44.6638893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:44.6639505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:44.7392561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu02kqmd6 2022-09-27T15:52:44.7393574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu02kqmd6/_remote_module_non_scriptable.py 2022-09-27T15:52:44.7627630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7n3zxx_9 2022-09-27T15:52:44.7628538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7n3zxx_9/_remote_module_non_scriptable.py 2022-09-27T15:52:44.7656712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8xzjvmgh 2022-09-27T15:52:44.7658128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8xzjvmgh/_remote_module_non_scriptable.py 2022-09-27T15:52:44.7901952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpseo73fot 2022-09-27T15:52:44.7903222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpseo73fot/_remote_module_non_scriptable.py 2022-09-27T15:52:44.9949715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:52:45.0242235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:52:45.0355503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:52:45.0469178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:52:45.6456877Z ok (2.457s) 2022-09-27T15:52:45.6457128Z 2022-09-27T15:52:45.6457617Z ---------------------------------------------------------------------- 2022-09-27T15:52:45.6457869Z Ran 1 test in 2.458s 2022-09-27T15:52:45.6457987Z 2022-09-27T15:52:45.6458068Z OK 2022-09-27T15:52:45.6458161Z 2022-09-27T15:52:45.6458242Z Generating XML reports... 2022-09-27T15:52:45.6493328Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155243.xml 2022-09-27T15:52:47.0124858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:47.0125256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:47.0135186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:47.0135560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:47.1743060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ym9f2ri 2022-09-27T15:52:47.1743858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ym9f2ri/_remote_module_non_scriptable.py 2022-09-27T15:52:47.4363748Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:52:47.4373199Z 2022-09-27T15:52:47.4373331Z Running tests... 2022-09-27T15:52:47.4373742Z ---------------------------------------------------------------------- 2022-09-27T15:52:47.6818499Z test_rpc_return_rref (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3606 2022-09-27T15:52:47.6846912Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3607 2022-09-27T15:52:47.6874390Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3608 2022-09-27T15:52:47.6904129Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3609 2022-09-27T15:52:48.8438489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:48.8439137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:48.8449818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:48.8450415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:48.8462335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:48.8462954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:48.8540692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:48.8541312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:48.8818563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:48.8819302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:48.8834717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:48.8835429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:48.8847938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:48.8848499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:48.8916130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:48.8916745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:49.0073787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvv_dz7mx 2022-09-27T15:52:49.0075230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvv_dz7mx/_remote_module_non_scriptable.py 2022-09-27T15:52:49.0097832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpktw39wk5 2022-09-27T15:52:49.0099922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpktw39wk5/_remote_module_non_scriptable.py 2022-09-27T15:52:49.0168174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyiq5out 2022-09-27T15:52:49.0169709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyiq5out/_remote_module_non_scriptable.py 2022-09-27T15:52:49.0281890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0wd52ss 2022-09-27T15:52:49.0283424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0wd52ss/_remote_module_non_scriptable.py 2022-09-27T15:52:49.2641657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:52:49.2662286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:52:49.2721034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:52:49.2827205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:52:49.9961628Z ok (2.558s) 2022-09-27T15:52:49.9961918Z 2022-09-27T15:52:49.9962409Z ---------------------------------------------------------------------- 2022-09-27T15:52:49.9962660Z Ran 1 test in 2.559s 2022-09-27T15:52:49.9962764Z 2022-09-27T15:52:49.9962826Z OK 2022-09-27T15:52:49.9962919Z 2022-09-27T15:52:49.9963018Z Generating XML reports... 2022-09-27T15:52:49.9997894Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155247.xml 2022-09-27T15:52:51.3573797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:51.3574178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:51.3581612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:51.3581974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:51.5144687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkylud78o 2022-09-27T15:52:51.5146187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkylud78o/_remote_module_non_scriptable.py 2022-09-27T15:52:51.7735765Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:52:51.7745238Z 2022-09-27T15:52:51.7745669Z Running tests... 2022-09-27T15:52:51.7746280Z ---------------------------------------------------------------------- 2022-09-27T15:52:52.0184243Z test_rpc_timeouts (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3829 2022-09-27T15:52:52.0211116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3830 2022-09-27T15:52:52.0239015Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3831 2022-09-27T15:52:52.0269007Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3832 2022-09-27T15:52:53.1007061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:53.1007666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:53.1039629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:53.1040315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:53.1399866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:53.1400625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:53.1435167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:53.1435792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:53.1462228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:53.1462846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:53.1665692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:52:53.1666337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:52:53.1809493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:53.1810145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:53.2014579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:52:53.2015258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:52:53.2659283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2n_e_v1u 2022-09-27T15:52:53.2660056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2n_e_v1u/_remote_module_non_scriptable.py 2022-09-27T15:52:53.2699461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiu8exvup 2022-09-27T15:52:53.2701129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiu8exvup/_remote_module_non_scriptable.py 2022-09-27T15:52:53.3063416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd0lutwkz 2022-09-27T15:52:53.3064886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd0lutwkz/_remote_module_non_scriptable.py 2022-09-27T15:52:53.3281804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mso543p 2022-09-27T15:52:53.3282572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mso543p/_remote_module_non_scriptable.py 2022-09-27T15:52:53.5226317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:52:53.5285261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:52:53.5637062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:52:53.5863229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:00.4422661Z ok (8.667s) 2022-09-27T15:53:00.4423157Z 2022-09-27T15:53:00.4423487Z ---------------------------------------------------------------------- 2022-09-27T15:53:00.4423808Z Ran 1 test in 8.668s 2022-09-27T15:53:00.4423927Z 2022-09-27T15:53:00.4423987Z OK 2022-09-27T15:53:00.4424065Z 2022-09-27T15:53:00.4424162Z Generating XML reports... 2022-09-27T15:53:00.4461001Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155251.xml 2022-09-27T15:53:01.8024411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:01.8024789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:01.8032430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:01.8033010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:01.9581802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpip4gsf2j 2022-09-27T15:53:01.9582843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpip4gsf2j/_remote_module_non_scriptable.py 2022-09-27T15:53:02.2242095Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:53:02.2251607Z 2022-09-27T15:53:02.2251716Z Running tests... 2022-09-27T15:53:02.2252310Z ---------------------------------------------------------------------- 2022-09-27T15:53:02.4789411Z test_rref_context_debug_info (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4052 2022-09-27T15:53:02.4817269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4053 2022-09-27T15:53:02.4845725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4054 2022-09-27T15:53:02.4875102Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4055 2022-09-27T15:53:03.6379092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:03.6379724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:03.6390203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:03.6402139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:03.6403017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:03.6403881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:03.6646303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:03.6646995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:03.6782638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:03.6783173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:03.6798265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:03.6798803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:03.6849875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:03.6850505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:03.7030351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:03.7031005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:03.8085042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphao32ba6 2022-09-27T15:53:03.8085789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphao32ba6/_remote_module_non_scriptable.py 2022-09-27T15:53:03.8086537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj0p6b6br 2022-09-27T15:53:03.8088007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj0p6b6br/_remote_module_non_scriptable.py 2022-09-27T15:53:03.8170415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9fmin68 2022-09-27T15:53:03.8171908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9fmin68/_remote_module_non_scriptable.py 2022-09-27T15:53:03.8285522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9pcihpi 2022-09-27T15:53:03.8286780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9pcihpi/_remote_module_non_scriptable.py 2022-09-27T15:53:04.0635334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:53:04.0683097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:53:04.0709432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:53:04.0838459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:04.3507675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:53:04.3609388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:53:04.3711504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:53:04.3712942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:53:04.3713814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:53:04.3715076Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:53:04.3716224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:53:04.3722513Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:53:04.9935477Z ok (2.768s) 2022-09-27T15:53:04.9935648Z 2022-09-27T15:53:04.9935975Z ---------------------------------------------------------------------- 2022-09-27T15:53:04.9936251Z Ran 1 test in 2.768s 2022-09-27T15:53:04.9936399Z 2022-09-27T15:53:04.9936478Z OK 2022-09-27T15:53:04.9936571Z 2022-09-27T15:53:04.9936651Z Generating XML reports... 2022-09-27T15:53:04.9972188Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155302.xml 2022-09-27T15:53:06.3644757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:06.3645133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:06.3652480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:06.3652995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:06.5225452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmki7n12 2022-09-27T15:53:06.5225957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmki7n12/_remote_module_non_scriptable.py 2022-09-27T15:53:06.7820131Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:53:06.7829763Z 2022-09-27T15:53:06.7829890Z Running tests... 2022-09-27T15:53:06.7830490Z ---------------------------------------------------------------------- 2022-09-27T15:53:07.0258573Z test_rref_forward_chain (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4287 2022-09-27T15:53:07.0286630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4288 2022-09-27T15:53:07.0313698Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4289 2022-09-27T15:53:07.0344223Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4290 2022-09-27T15:53:08.1552832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:08.1553456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:08.1698513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:08.1699116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:08.1898115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:08.1898764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:08.2057417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:08.2058085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:08.2243011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:08.2243681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:08.2317926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:08.2318566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:08.2614802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:08.2615438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:08.2681361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:08.2681994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:08.3162589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmv53w9s 2022-09-27T15:53:08.3163394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmv53w9s/_remote_module_non_scriptable.py 2022-09-27T15:53:08.3327278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3f631oc1 2022-09-27T15:53:08.3328063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3f631oc1/_remote_module_non_scriptable.py 2022-09-27T15:53:08.3881054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3gaas434 2022-09-27T15:53:08.3881806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3gaas434/_remote_module_non_scriptable.py 2022-09-27T15:53:08.3939214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3q396ge 2022-09-27T15:53:08.3940912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3q396ge/_remote_module_non_scriptable.py 2022-09-27T15:53:08.5711056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:53:08.5881688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:53:08.6471928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:08.6516868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:53:09.3401883Z ok (2.557s) 2022-09-27T15:53:09.3402123Z 2022-09-27T15:53:09.3402437Z ---------------------------------------------------------------------- 2022-09-27T15:53:09.3402933Z Ran 1 test in 2.557s 2022-09-27T15:53:09.3403054Z 2022-09-27T15:53:09.3403115Z OK 2022-09-27T15:53:09.3403208Z 2022-09-27T15:53:09.3403362Z Generating XML reports... 2022-09-27T15:53:09.3437640Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155306.xml 2022-09-27T15:53:10.7219232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:10.7219610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:10.7226490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:10.7226854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:10.8799051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpymegrblf 2022-09-27T15:53:10.8800157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpymegrblf/_remote_module_non_scriptable.py 2022-09-27T15:53:11.1422857Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:53:11.1432388Z 2022-09-27T15:53:11.1432490Z Running tests... 2022-09-27T15:53:11.1433344Z ---------------------------------------------------------------------- 2022-09-27T15:53:11.3872022Z test_rref_get_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4510 2022-09-27T15:53:11.3899415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4511 2022-09-27T15:53:11.3926578Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4512 2022-09-27T15:53:11.3955772Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4513 2022-09-27T15:53:12.4836976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:12.4837604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:12.4959008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:12.4959753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:12.4989138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:12.4989794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:12.5230228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:12.5230928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:12.5301914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:12.5302623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:12.5349654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:12.5350316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:12.5375510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:12.5376118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:12.5655771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:12.5656428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:12.6500243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6zvhkxk 2022-09-27T15:53:12.6501129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6zvhkxk/_remote_module_non_scriptable.py 2022-09-27T15:53:12.6615722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuhymk67w 2022-09-27T15:53:12.6616670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuhymk67w/_remote_module_non_scriptable.py 2022-09-27T15:53:12.6641358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxgi2cj7 2022-09-27T15:53:12.6642702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxgi2cj7/_remote_module_non_scriptable.py 2022-09-27T15:53:12.6930774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yhtimzp 2022-09-27T15:53:12.6931575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yhtimzp/_remote_module_non_scriptable.py 2022-09-27T15:53:12.9088226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:53:12.9182915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:53:12.9217086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:12.9530294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:53:13.6011487Z ok (2.458s) 2022-09-27T15:53:13.6011711Z 2022-09-27T15:53:13.6012134Z ---------------------------------------------------------------------- 2022-09-27T15:53:13.6012534Z Ran 1 test in 2.458s 2022-09-27T15:53:13.6012745Z 2022-09-27T15:53:13.6012853Z OK 2022-09-27T15:53:13.6013022Z 2022-09-27T15:53:13.6013163Z Generating XML reports... 2022-09-27T15:53:13.6049008Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155311.xml 2022-09-27T15:53:14.9816837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:14.9817282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:14.9825072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:14.9825480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:15.1388726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbvcdci2f 2022-09-27T15:53:15.1389589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbvcdci2f/_remote_module_non_scriptable.py 2022-09-27T15:53:15.4012277Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:53:15.4021911Z 2022-09-27T15:53:15.4022048Z Running tests... 2022-09-27T15:53:15.4022430Z ---------------------------------------------------------------------- 2022-09-27T15:53:15.6429884Z test_rref_leak (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4733 2022-09-27T15:53:15.6459263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4734 2022-09-27T15:53:15.6487126Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4735 2022-09-27T15:53:15.6517747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4736 2022-09-27T15:53:16.7286009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:16.7286633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:16.7638640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:16.7639328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:16.7717044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:16.7717833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:16.7718897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:16.7719544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:16.7739814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:16.7740450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:16.8095346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:16.8095969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:16.8096822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:16.8097442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:16.8114663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:16.8115331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:16.8905957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3hz71pl 2022-09-27T15:53:16.8906796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3hz71pl/_remote_module_non_scriptable.py 2022-09-27T15:53:16.9370207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdsltrixj 2022-09-27T15:53:16.9370912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14g_5fd0 2022-09-27T15:53:16.9372299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdsltrixj/_remote_module_non_scriptable.py 2022-09-27T15:53:16.9373035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14g_5fd0/_remote_module_non_scriptable.py 2022-09-27T15:53:16.9374117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3gnjf04f 2022-09-27T15:53:16.9375252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3gnjf04f/_remote_module_non_scriptable.py 2022-09-27T15:53:17.1481441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:53:17.1937110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:17.1953501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:53:17.1954472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:53:17.4308086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:53:17.4407979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:53:17.4408813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:53:17.4409628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:53:17.4410875Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:53:17.4412058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:53:17.4418694Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:53:17.4419868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:53:17.4823650Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-09-27T15:53:17.4825567Z Leaking RRef GloballyUniqueId(created_on=1, local_id=0) with fork Id GloballyUniqueId(created_on=1, local_id=1) 2022-09-27T15:53:17.4826024Z 2022-09-27T15:53:17.4833341Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-09-27T15:53:17.4835432Z Leaking RRef GloballyUniqueId(created_on=3, local_id=0) with fork Id GloballyUniqueId(created_on=3, local_id=1) 2022-09-27T15:53:17.4835878Z 2022-09-27T15:53:17.4849538Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-09-27T15:53:17.4851144Z Leaking RRef GloballyUniqueId(created_on=0, local_id=0) with fork Id GloballyUniqueId(created_on=0, local_id=1) 2022-09-27T15:53:17.4851715Z 2022-09-27T15:53:17.4909232Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-09-27T15:53:17.4910365Z Leaking RRef GloballyUniqueId(created_on=2, local_id=0) with fork Id GloballyUniqueId(created_on=2, local_id=1) 2022-09-27T15:53:17.4910662Z 2022-09-27T15:53:17.8573296Z ok (2.455s) 2022-09-27T15:53:17.8573575Z 2022-09-27T15:53:17.8574071Z ---------------------------------------------------------------------- 2022-09-27T15:53:17.8574322Z Ran 1 test in 2.455s 2022-09-27T15:53:17.8574439Z 2022-09-27T15:53:17.8574498Z OK 2022-09-27T15:53:17.8574589Z 2022-09-27T15:53:17.8574684Z Generating XML reports... 2022-09-27T15:53:17.8611029Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155315.xml 2022-09-27T15:53:19.2397552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:19.2397986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:19.2405460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:19.2405910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:19.3991618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fh57yhb 2022-09-27T15:53:19.3992535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fh57yhb/_remote_module_non_scriptable.py 2022-09-27T15:53:19.6659004Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:53:19.6668767Z 2022-09-27T15:53:19.6669023Z Running tests... 2022-09-27T15:53:19.6669436Z ---------------------------------------------------------------------- 2022-09-27T15:53:19.9171685Z test_rref_proxy_class (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4968 2022-09-27T15:53:19.9199183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4969 2022-09-27T15:53:19.9226832Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4970 2022-09-27T15:53:19.9257248Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4971 2022-09-27T15:53:21.0500246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:21.0500862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:21.0501593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:21.0502162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:21.0516405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:21.0517044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:21.0555469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:21.0556153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:21.0896154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:21.0896782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:21.0897494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:21.0898014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:21.0913336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:21.0914178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:21.0942299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:21.0942871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:21.2169740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrzre554 2022-09-27T15:53:21.2170483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4sst45qs 2022-09-27T15:53:21.2171168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrzre554/_remote_module_non_scriptable.py 2022-09-27T15:53:21.2171851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4sst45qs/_remote_module_non_scriptable.py 2022-09-27T15:53:21.2188255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94_ihb9w 2022-09-27T15:53:21.2190079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94_ihb9w/_remote_module_non_scriptable.py 2022-09-27T15:53:21.2202127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkcuy0yoa 2022-09-27T15:53:21.2204543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkcuy0yoa/_remote_module_non_scriptable.py 2022-09-27T15:53:21.4761144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:53:21.4776126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:53:21.4815174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:21.4940035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:53:22.2316043Z ok (2.564s) 2022-09-27T15:53:22.2316281Z 2022-09-27T15:53:22.2316639Z ---------------------------------------------------------------------- 2022-09-27T15:53:22.2316954Z Ran 1 test in 2.565s 2022-09-27T15:53:22.2317071Z 2022-09-27T15:53:22.2317134Z OK 2022-09-27T15:53:22.2317212Z 2022-09-27T15:53:22.2317557Z Generating XML reports... 2022-09-27T15:53:22.2352464Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155319.xml 2022-09-27T15:53:23.6141274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:23.6141891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:23.6150114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:23.6150746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:23.7710266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwl8kgdsm 2022-09-27T15:53:23.7711309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwl8kgdsm/_remote_module_non_scriptable.py 2022-09-27T15:53:24.0321466Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:53:24.0330459Z 2022-09-27T15:53:24.0330595Z Running tests... 2022-09-27T15:53:24.0331206Z ---------------------------------------------------------------------- 2022-09-27T15:53:24.2761431Z test_rref_proxy_class_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5191 2022-09-27T15:53:24.2789077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5192 2022-09-27T15:53:24.2816639Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5193 2022-09-27T15:53:24.2845641Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5194 2022-09-27T15:53:25.4242527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:25.4243151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:25.4617115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:25.4617726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:25.4742017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:25.4742665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:25.4843049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:25.4843769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:25.4947859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:25.4948546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:25.5117521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:25.5118120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:25.5218710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:25.5219355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:25.5314344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:25.5315162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:25.5895854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgzffsnc 2022-09-27T15:53:25.5896674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgzffsnc/_remote_module_non_scriptable.py 2022-09-27T15:53:25.6383644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6wj9n1pd 2022-09-27T15:53:25.6384704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6wj9n1pd/_remote_module_non_scriptable.py 2022-09-27T15:53:25.6479880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvh9hh5ey 2022-09-27T15:53:25.6481464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvh9hh5ey/_remote_module_non_scriptable.py 2022-09-27T15:53:25.6574197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmyh5p78 2022-09-27T15:53:25.6575582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmyh5p78/_remote_module_non_scriptable.py 2022-09-27T15:53:25.8475911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:53:25.8938332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:53:25.9049725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:53:25.9150628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:26.5904330Z ok (2.557s) 2022-09-27T15:53:26.5904599Z 2022-09-27T15:53:26.5905109Z ---------------------------------------------------------------------- 2022-09-27T15:53:26.5905541Z Ran 1 test in 2.557s 2022-09-27T15:53:26.5905727Z 2022-09-27T15:53:26.5905840Z OK 2022-09-27T15:53:26.5905999Z 2022-09-27T15:53:26.5906163Z Generating XML reports... 2022-09-27T15:53:26.5941731Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155324.xml 2022-09-27T15:53:27.9563192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:27.9563569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:27.9571071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:27.9571433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:28.1138946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcycfqk2s 2022-09-27T15:53:28.1140388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcycfqk2s/_remote_module_non_scriptable.py 2022-09-27T15:53:28.3780607Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:53:28.3790711Z 2022-09-27T15:53:28.3790827Z Running tests... 2022-09-27T15:53:28.3791347Z ---------------------------------------------------------------------- 2022-09-27T15:53:28.6237515Z test_rref_proxy_non_exist (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5414 2022-09-27T15:53:28.6264462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5415 2022-09-27T15:53:28.6291658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5416 2022-09-27T15:53:28.6321161Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5417 2022-09-27T15:53:29.7095701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:29.7096389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:29.7330836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:29.7331336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:29.7459572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:29.7460226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:29.7532028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:29.7532942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:29.7695659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:29.7696583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:29.7699454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:29.7700158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:29.7897729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:29.7898406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:29.8048221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:29.8048896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:29.8735705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyti_tg3 2022-09-27T15:53:29.8736546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyti_tg3/_remote_module_non_scriptable.py 2022-09-27T15:53:29.8966937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprw4jlq8h 2022-09-27T15:53:29.8967843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprw4jlq8h/_remote_module_non_scriptable.py 2022-09-27T15:53:29.9165308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprp38guf8 2022-09-27T15:53:29.9166106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprp38guf8/_remote_module_non_scriptable.py 2022-09-27T15:53:29.9300339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumjmu4aj 2022-09-27T15:53:29.9301279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumjmu4aj/_remote_module_non_scriptable.py 2022-09-27T15:53:30.1314117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:53:30.1524913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:30.1722396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:53:30.1857325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:53:30.8376916Z ok (2.458s) 2022-09-27T15:53:30.8377203Z 2022-09-27T15:53:30.8377704Z ---------------------------------------------------------------------- 2022-09-27T15:53:30.8377968Z Ran 1 test in 2.459s 2022-09-27T15:53:30.8378086Z 2022-09-27T15:53:30.8378146Z OK 2022-09-27T15:53:30.8378239Z 2022-09-27T15:53:30.8378334Z Generating XML reports... 2022-09-27T15:53:30.8413380Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155328.xml 2022-09-27T15:53:32.2266166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:32.2266551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:32.2275317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:32.2275677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:32.3917691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyj0xg6s3 2022-09-27T15:53:32.3918644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj0xg6s3/_remote_module_non_scriptable.py 2022-09-27T15:53:32.6542421Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:53:32.6551480Z 2022-09-27T15:53:32.6551603Z Running tests... 2022-09-27T15:53:32.6552203Z ---------------------------------------------------------------------- 2022-09-27T15:53:32.9026052Z test_rref_proxy_reuse (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5637 2022-09-27T15:53:32.9054263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5638 2022-09-27T15:53:32.9082888Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5639 2022-09-27T15:53:32.9112395Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5640 2022-09-27T15:53:34.0082447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:34.0083124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:34.0096351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:34.0096980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:34.0117474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:34.0118265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:34.0340362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:34.0340853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:34.0462562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:34.0463319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:34.0481726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:34.0482434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:34.0500936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:34.0501598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:34.0707221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:34.0707830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:34.1725558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxlo2cigl 2022-09-27T15:53:34.1727040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxlo2cigl/_remote_module_non_scriptable.py 2022-09-27T15:53:34.1736674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1zfwj5x_ 2022-09-27T15:53:34.1738585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1zfwj5x_/_remote_module_non_scriptable.py 2022-09-27T15:53:34.1760439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_twkp83w 2022-09-27T15:53:34.1762330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_twkp83w/_remote_module_non_scriptable.py 2022-09-27T15:53:34.1969773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqd3mva5t 2022-09-27T15:53:34.1970533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqd3mva5t/_remote_module_non_scriptable.py 2022-09-27T15:53:34.4260647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:53:34.4276192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:53:34.4326183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:53:34.4544683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:35.1169622Z ok (2.461s) 2022-09-27T15:53:35.1169874Z 2022-09-27T15:53:35.1170307Z ---------------------------------------------------------------------- 2022-09-27T15:53:35.1170818Z Ran 1 test in 2.462s 2022-09-27T15:53:35.1170940Z 2022-09-27T15:53:35.1171001Z OK 2022-09-27T15:53:35.1171104Z 2022-09-27T15:53:35.1171237Z Generating XML reports... 2022-09-27T15:53:35.1204838Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155332.xml 2022-09-27T15:53:36.4943988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:36.4944371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:36.4952068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:36.4952481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:36.6505577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7b0xhfw 2022-09-27T15:53:36.6506578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7b0xhfw/_remote_module_non_scriptable.py 2022-09-27T15:53:36.9162102Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:53:36.9172228Z 2022-09-27T15:53:36.9172722Z Running tests... 2022-09-27T15:53:36.9173119Z ---------------------------------------------------------------------- 2022-09-27T15:53:37.1608297Z test_rref_proxy_tensor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5860 2022-09-27T15:53:37.1636095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5861 2022-09-27T15:53:37.1664219Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5862 2022-09-27T15:53:37.1693543Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5863 2022-09-27T15:53:38.2730342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:38.2730953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:38.2733651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:38.2734281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:38.2773205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:38.2773858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:38.2930332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:38.2930861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:38.3107187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:38.3107753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:38.3115523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:38.3116097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:38.3150772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:38.3151386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:38.3301418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:38.3302062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:38.4376007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tjy08j4 2022-09-27T15:53:38.4377146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tjy08j4/_remote_module_non_scriptable.py 2022-09-27T15:53:38.4377999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9jj5o_h 2022-09-27T15:53:38.4381006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9jj5o_h/_remote_module_non_scriptable.py 2022-09-27T15:53:38.4404475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg5hbchbt 2022-09-27T15:53:38.4405888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg5hbchbt/_remote_module_non_scriptable.py 2022-09-27T15:53:38.4568876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbd_o5m8 2022-09-27T15:53:38.4570295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbd_o5m8/_remote_module_non_scriptable.py 2022-09-27T15:53:38.6911115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:53:38.6912214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:53:38.6947298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:38.7130588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:53:39.4753401Z ok (2.558s) 2022-09-27T15:53:39.4753680Z 2022-09-27T15:53:39.4754187Z ---------------------------------------------------------------------- 2022-09-27T15:53:39.4754455Z Ran 1 test in 2.558s 2022-09-27T15:53:39.4754720Z 2022-09-27T15:53:39.4754780Z OK 2022-09-27T15:53:39.4754873Z 2022-09-27T15:53:39.4754969Z Generating XML reports... 2022-09-27T15:53:39.4791094Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155336.xml 2022-09-27T15:53:40.8638016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:40.8638387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:40.8646888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:40.8647270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:41.0270612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3eoh601g 2022-09-27T15:53:41.0271125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3eoh601g/_remote_module_non_scriptable.py 2022-09-27T15:53:41.2917627Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:53:41.2927490Z 2022-09-27T15:53:41.2927623Z Running tests... 2022-09-27T15:53:41.2928043Z ---------------------------------------------------------------------- 2022-09-27T15:53:41.5383886Z test_rref_proxy_tensor_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6083 2022-09-27T15:53:41.5411721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6084 2022-09-27T15:53:41.5439620Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6085 2022-09-27T15:53:41.5469415Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6086 2022-09-27T15:53:42.6307277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:42.6307884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:42.6549761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:42.6550342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:42.6675740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:42.6676335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:42.6819546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:42.6820260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:42.6836958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:42.6837574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:42.6931364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:42.6931943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:42.7191683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:42.7192422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:42.7210658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:42.7211308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:42.7949765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppt9hwd1h 2022-09-27T15:53:42.7950795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppt9hwd1h/_remote_module_non_scriptable.py 2022-09-27T15:53:42.8219237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzf2fe23h 2022-09-27T15:53:42.8220080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzf2fe23h/_remote_module_non_scriptable.py 2022-09-27T15:53:42.8461524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyk5yuzn6 2022-09-27T15:53:42.8462408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyk5yuzn6/_remote_module_non_scriptable.py 2022-09-27T15:53:42.8480291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4svpej8w 2022-09-27T15:53:42.8482536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4svpej8w/_remote_module_non_scriptable.py 2022-09-27T15:53:43.0557000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:53:43.0851164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:53:43.1062936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:53:43.1243484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:43.7525952Z ok (2.459s) 2022-09-27T15:53:43.7526240Z 2022-09-27T15:53:43.7526700Z ---------------------------------------------------------------------- 2022-09-27T15:53:43.7526954Z Ran 1 test in 2.460s 2022-09-27T15:53:43.7527061Z 2022-09-27T15:53:43.7527120Z OK 2022-09-27T15:53:43.7527212Z 2022-09-27T15:53:43.7527305Z Generating XML reports... 2022-09-27T15:53:43.7563835Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155341.xml 2022-09-27T15:53:45.1229109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:45.1229776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:45.1238370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:45.1238957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:45.2786868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedjme_94 2022-09-27T15:53:45.2787882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedjme_94/_remote_module_non_scriptable.py 2022-09-27T15:53:45.5387694Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:53:45.5397689Z 2022-09-27T15:53:45.5398019Z Running tests... 2022-09-27T15:53:45.5398900Z ---------------------------------------------------------------------- 2022-09-27T15:53:45.7781831Z test_rref_py_pickle_not_supported (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6306 2022-09-27T15:53:45.7810359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6307 2022-09-27T15:53:45.7839557Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6308 2022-09-27T15:53:45.7870871Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6309 2022-09-27T15:53:46.9017273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:46.9017910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:46.9381270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:46.9381902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:46.9429352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:46.9429952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:46.9450776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:46.9451407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:46.9558159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:46.9558664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:46.9812579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:46.9813316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:46.9830313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:46.9830932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:46.9934083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:46.9934734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:47.0660527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kz17z6r 2022-09-27T15:53:47.0661307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kz17z6r/_remote_module_non_scriptable.py 2022-09-27T15:53:47.1089880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bkygaka 2022-09-27T15:53:47.1090685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bkygaka/_remote_module_non_scriptable.py 2022-09-27T15:53:47.1095280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpezjbti8n 2022-09-27T15:53:47.1097101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpezjbti8n/_remote_module_non_scriptable.py 2022-09-27T15:53:47.1188154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1tiyf75s 2022-09-27T15:53:47.1189546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1tiyf75s/_remote_module_non_scriptable.py 2022-09-27T15:53:47.3219870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:53:47.3674957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:47.3728092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:53:47.3852841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:53:47.9925391Z ok (2.452s) 2022-09-27T15:53:47.9925547Z 2022-09-27T15:53:47.9926734Z ---------------------------------------------------------------------- 2022-09-27T15:53:47.9927046Z Ran 1 test in 2.453s 2022-09-27T15:53:47.9927238Z 2022-09-27T15:53:47.9927300Z OK 2022-09-27T15:53:47.9927402Z 2022-09-27T15:53:47.9927483Z Generating XML reports... 2022-09-27T15:53:47.9962135Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155345.xml 2022-09-27T15:53:49.3675871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:49.3676478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:49.3684879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:49.3685529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:49.5231691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp731pdhhz 2022-09-27T15:53:49.5232244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp731pdhhz/_remote_module_non_scriptable.py 2022-09-27T15:53:49.7828052Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:53:49.7838087Z 2022-09-27T15:53:49.7838234Z Running tests... 2022-09-27T15:53:49.7838841Z ---------------------------------------------------------------------- 2022-09-27T15:53:50.0303157Z test_rref_str (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6529 2022-09-27T15:53:50.0329792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6530 2022-09-27T15:53:50.0357732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6531 2022-09-27T15:53:50.0388268Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6532 2022-09-27T15:53:51.1959996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:51.1960677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:51.2054534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:51.2055110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:51.2073358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:51.2073996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:51.2339402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:51.2340107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:51.2369691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:51.2370288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:51.2437541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:51.2438221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:51.2455068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:51.2455613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:51.2748999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:51.2749636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:51.3649674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjto0a4f8 2022-09-27T15:53:51.3650701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjto0a4f8/_remote_module_non_scriptable.py 2022-09-27T15:53:51.3718464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57ae822a 2022-09-27T15:53:51.3720190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57ae822a/_remote_module_non_scriptable.py 2022-09-27T15:53:51.3909815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpht3dagla 2022-09-27T15:53:51.3911063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpht3dagla/_remote_module_non_scriptable.py 2022-09-27T15:53:51.4041970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8u9lny69 2022-09-27T15:53:51.4043085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8u9lny69/_remote_module_non_scriptable.py 2022-09-27T15:53:51.6213086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:53:51.6263741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:51.6458037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:53:51.6605501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:53:52.3446240Z ok (2.560s) 2022-09-27T15:53:52.3446499Z 2022-09-27T15:53:52.3446900Z ---------------------------------------------------------------------- 2022-09-27T15:53:52.3447156Z Ran 1 test in 2.561s 2022-09-27T15:53:52.3447261Z 2022-09-27T15:53:52.3447329Z OK 2022-09-27T15:53:52.3447422Z 2022-09-27T15:53:52.3447518Z Generating XML reports... 2022-09-27T15:53:52.3485652Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155349.xml 2022-09-27T15:53:53.7133071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:53.7133736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:53.7141907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:53.7142525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:53.8698497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitdr9ulk 2022-09-27T15:53:53.8699392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitdr9ulk/_remote_module_non_scriptable.py 2022-09-27T15:53:54.1294585Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:53:54.1304665Z 2022-09-27T15:53:54.1304959Z Running tests... 2022-09-27T15:53:54.1305542Z ---------------------------------------------------------------------- 2022-09-27T15:53:54.3757585Z test_rref_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6752 2022-09-27T15:53:54.3784263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6753 2022-09-27T15:53:54.3813810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6754 2022-09-27T15:53:54.3843401Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6755 2022-09-27T15:53:55.5337626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:55.5338303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:55.5499628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:55.5500209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:55.5714771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:55.5715452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:55.5732639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:55.5733401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:55.5858779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:55.5859380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:55.5882566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:55.5883120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:55.6121254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:55.6121874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:55.6244798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:55.6245431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:53:55.6991171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4y43_6qe 2022-09-27T15:53:55.6992049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4y43_6qe/_remote_module_non_scriptable.py 2022-09-27T15:53:55.7164352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcea1isjs 2022-09-27T15:53:55.7165220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcea1isjs/_remote_module_non_scriptable.py 2022-09-27T15:53:55.7391807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3quy9nkz 2022-09-27T15:53:55.7392823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3quy9nkz/_remote_module_non_scriptable.py 2022-09-27T15:53:55.7531270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcm5szok 2022-09-27T15:53:55.7532051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcm5szok/_remote_module_non_scriptable.py 2022-09-27T15:53:55.9611625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:53:55.9797653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:53:56.0030847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:53:56.0168339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:53:58.5933598Z ok (4.462s) 2022-09-27T15:53:58.5933833Z 2022-09-27T15:53:58.5934359Z ---------------------------------------------------------------------- 2022-09-27T15:53:58.5934728Z Ran 1 test in 4.463s 2022-09-27T15:53:58.5934852Z 2022-09-27T15:53:58.5934899Z OK 2022-09-27T15:53:58.5934994Z 2022-09-27T15:53:58.5935088Z Generating XML reports... 2022-09-27T15:53:58.5969849Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155354.xml 2022-09-27T15:53:59.9672252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:53:59.9672633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:53:59.9679932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:53:59.9680297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:00.1252999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_kpy185 2022-09-27T15:54:00.1254047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_kpy185/_remote_module_non_scriptable.py 2022-09-27T15:54:00.3901911Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:00.3911989Z 2022-09-27T15:54:00.3912144Z Running tests... 2022-09-27T15:54:00.3912615Z ---------------------------------------------------------------------- 2022-09-27T15:54:00.6323731Z test_rref_type_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6975 2022-09-27T15:54:00.6351710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6976 2022-09-27T15:54:00.6380031Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6977 2022-09-27T15:54:00.6409488Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6978 2022-09-27T15:54:01.8259959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:01.8260670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:01.8322913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:01.8323481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:01.8562174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:01.8562823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:01.8641647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:01.8642249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:01.8709669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:01.8944676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:01.8945691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:01.8946331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:01.9018569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:01.9019185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:01.9411912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:01.9412585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:01.9932113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42yjinru 2022-09-27T15:54:01.9932996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42yjinru/_remote_module_non_scriptable.py 2022-09-27T15:54:01.9988171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvrh7vpf 2022-09-27T15:54:01.9989266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvrh7vpf/_remote_module_non_scriptable.py 2022-09-27T15:54:02.0217724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6zdpx3y 2022-09-27T15:54:02.0218509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6zdpx3y/_remote_module_non_scriptable.py 2022-09-27T15:54:02.0681029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqqmzpg7 2022-09-27T15:54:02.0681976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqqmzpg7/_remote_module_non_scriptable.py 2022-09-27T15:54:02.2527936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:54:02.2587022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:02.2829210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:02.3282293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:03.0470915Z ok (2.656s) 2022-09-27T15:54:03.0471249Z 2022-09-27T15:54:03.0471773Z ---------------------------------------------------------------------- 2022-09-27T15:54:03.0472153Z Ran 1 test in 2.656s 2022-09-27T15:54:03.0472271Z 2022-09-27T15:54:03.0472332Z OK 2022-09-27T15:54:03.0472426Z 2022-09-27T15:54:03.0472516Z Generating XML reports... 2022-09-27T15:54:03.0509524Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155400.xml 2022-09-27T15:54:04.4426035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:04.4426424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:04.4434766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:04.4435127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:04.5984543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz44f3o9m 2022-09-27T15:54:04.5985570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz44f3o9m/_remote_module_non_scriptable.py 2022-09-27T15:54:04.8572829Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:04.8582036Z 2022-09-27T15:54:04.8582123Z Running tests... 2022-09-27T15:54:04.8582710Z ---------------------------------------------------------------------- 2022-09-27T15:54:05.1010423Z test_rref_type_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7198 2022-09-27T15:54:05.1037682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7199 2022-09-27T15:54:05.1065531Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7200 2022-09-27T15:54:05.1094637Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7201 2022-09-27T15:54:06.1861220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:06.1861922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:06.1962875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:06.1963455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:06.2245650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:06.2246305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:06.2252544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:06.2253339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:06.2356477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:06.2357157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:06.2387324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:06.2387947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:06.2631510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:06.2632060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:06.2762740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:06.2763402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:06.3513109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7rnnejh 2022-09-27T15:54:06.3513894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7rnnejh/_remote_module_non_scriptable.py 2022-09-27T15:54:06.3630184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgtyxz0iv 2022-09-27T15:54:06.3631744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgtyxz0iv/_remote_module_non_scriptable.py 2022-09-27T15:54:06.3900619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0036kpi 2022-09-27T15:54:06.3901409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0036kpi/_remote_module_non_scriptable.py 2022-09-27T15:54:06.4050621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwiyhqxi 2022-09-27T15:54:06.4051784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwiyhqxi/_remote_module_non_scriptable.py 2022-09-27T15:54:06.6093857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:06.6212562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:54:06.6510786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:06.6657157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:07.3151107Z ok (2.457s) 2022-09-27T15:54:07.3151363Z 2022-09-27T15:54:07.3151714Z ---------------------------------------------------------------------- 2022-09-27T15:54:07.3152103Z Ran 1 test in 2.457s 2022-09-27T15:54:07.3152320Z 2022-09-27T15:54:07.3152444Z OK 2022-09-27T15:54:07.3152615Z 2022-09-27T15:54:07.3152728Z Generating XML reports... 2022-09-27T15:54:07.3188560Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155404.xml 2022-09-27T15:54:08.6890028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:08.6890409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:08.6900156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:08.6900515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:08.8533549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe225n8i6 2022-09-27T15:54:08.8534639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe225n8i6/_remote_module_non_scriptable.py 2022-09-27T15:54:09.1205284Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:09.1214692Z 2022-09-27T15:54:09.1214827Z Running tests... 2022-09-27T15:54:09.1215395Z ---------------------------------------------------------------------- 2022-09-27T15:54:09.3729538Z test_rref_type_owner_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7421 2022-09-27T15:54:09.3757525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7422 2022-09-27T15:54:09.3786601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7423 2022-09-27T15:54:09.3817612Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7424 2022-09-27T15:54:10.5489524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:10.5490126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:10.5651777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:10.5652450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:10.5854897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:10.5855814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:10.6005720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:10.6006699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:10.6022402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:10.6023023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:10.6147607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:10.6148276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:10.6385783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:10.6386345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:10.6527873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:10.6528508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:10.7133494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7gf91li 2022-09-27T15:54:10.7134308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7gf91li/_remote_module_non_scriptable.py 2022-09-27T15:54:10.7295142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdy7_dicm 2022-09-27T15:54:10.7295829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdy7_dicm/_remote_module_non_scriptable.py 2022-09-27T15:54:10.7669411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9pvzdz9 2022-09-27T15:54:10.7670745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9pvzdz9/_remote_module_non_scriptable.py 2022-09-27T15:54:10.7800916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw396ozcg 2022-09-27T15:54:10.7801947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw396ozcg/_remote_module_non_scriptable.py 2022-09-27T15:54:10.9709331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:10.9880559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:11.0286020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:54:11.0367515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:11.6876320Z ok (2.566s) 2022-09-27T15:54:11.6876568Z 2022-09-27T15:54:11.6877097Z ---------------------------------------------------------------------- 2022-09-27T15:54:11.6877460Z Ran 1 test in 2.566s 2022-09-27T15:54:11.6877564Z 2022-09-27T15:54:11.6877638Z OK 2022-09-27T15:54:11.6877731Z 2022-09-27T15:54:11.6877828Z Generating XML reports... 2022-09-27T15:54:11.6912391Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155409.xml 2022-09-27T15:54:13.0969590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:13.0970015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:13.0979419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:13.0979768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:13.2587629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp208q6oui 2022-09-27T15:54:13.2588613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp208q6oui/_remote_module_non_scriptable.py 2022-09-27T15:54:13.5172023Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:13.5181214Z 2022-09-27T15:54:13.5181464Z Running tests... 2022-09-27T15:54:13.5181850Z ---------------------------------------------------------------------- 2022-09-27T15:54:13.7625996Z test_rref_type_owner_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7644 2022-09-27T15:54:13.7653332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7645 2022-09-27T15:54:13.7681100Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7646 2022-09-27T15:54:13.7710020Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7647 2022-09-27T15:54:14.8621962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:14.8622647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:14.8951282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:14.8952018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:14.9024533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:14.9025134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:14.9344258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:14.9344929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:14.9381970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:14.9382691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:14.9405666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:14.9406299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:14.9783909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:14.9784575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:14.9802080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:14.9802744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:15.0323078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74uuddc9 2022-09-27T15:54:15.0324267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74uuddc9/_remote_module_non_scriptable.py 2022-09-27T15:54:15.0637437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzrnh_o9k 2022-09-27T15:54:15.0638299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzrnh_o9k/_remote_module_non_scriptable.py 2022-09-27T15:54:15.1077223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6rw_tx6p 2022-09-27T15:54:15.1078172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6rw_tx6p/_remote_module_non_scriptable.py 2022-09-27T15:54:15.1100790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzzogah78 2022-09-27T15:54:15.1103394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzzogah78/_remote_module_non_scriptable.py 2022-09-27T15:54:15.2956827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:15.3449979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:15.3702850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:15.3715402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:54:15.9767714Z ok (2.458s) 2022-09-27T15:54:15.9767992Z 2022-09-27T15:54:15.9768506Z ---------------------------------------------------------------------- 2022-09-27T15:54:15.9768879Z Ran 1 test in 2.459s 2022-09-27T15:54:15.9768999Z 2022-09-27T15:54:15.9769060Z OK 2022-09-27T15:54:15.9769140Z 2022-09-27T15:54:15.9769233Z Generating XML reports... 2022-09-27T15:54:15.9803482Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155413.xml 2022-09-27T15:54:17.3439555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:17.3439933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:17.3446800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:17.3447214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:17.5019491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkz69m31 2022-09-27T15:54:17.5020650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkz69m31/_remote_module_non_scriptable.py 2022-09-27T15:54:17.7630250Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:17.7640235Z 2022-09-27T15:54:17.7640411Z Running tests... 2022-09-27T15:54:17.7640752Z ---------------------------------------------------------------------- 2022-09-27T15:54:18.0022800Z test_rref_type_slow_init (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7867 2022-09-27T15:54:18.0049827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7868 2022-09-27T15:54:18.0077156Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7869 2022-09-27T15:54:18.0106427Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7870 2022-09-27T15:54:19.1247763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:19.1248449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:19.1337775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:19.1338276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:19.1346629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:19.1347299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:19.1645854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:19.1646527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:19.1661735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:19.1662456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:19.1731435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:19.1732030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:19.1742062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:19.1742615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:19.2049853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:19.2050748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:19.2923438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk5ija194 2022-09-27T15:54:19.2925068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk5ija194/_remote_module_non_scriptable.py 2022-09-27T15:54:19.2998918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2dofum3 2022-09-27T15:54:19.3000431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2dofum3/_remote_module_non_scriptable.py 2022-09-27T15:54:19.3001117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3r0e9_3m 2022-09-27T15:54:19.3003889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3r0e9_3m/_remote_module_non_scriptable.py 2022-09-27T15:54:19.3495991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8bd2ili 2022-09-27T15:54:19.3496685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8bd2ili/_remote_module_non_scriptable.py 2022-09-27T15:54:19.5497063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:19.5559444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:54:19.5568614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:19.6053947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:21.2179521Z ok (3.453s) 2022-09-27T15:54:21.2211120Z 2022-09-27T15:54:21.2212001Z ---------------------------------------------------------------------- 2022-09-27T15:54:21.2212462Z Ran 1 test in 3.454s 2022-09-27T15:54:21.2212610Z 2022-09-27T15:54:21.2212672Z OK 2022-09-27T15:54:21.2212767Z 2022-09-27T15:54:21.2212862Z Generating XML reports... 2022-09-27T15:54:21.2222715Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155417.xml 2022-09-27T15:54:22.6034381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:22.6034916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:22.6043342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:22.6043771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:22.7600052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4og0l93t 2022-09-27T15:54:22.7601047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4og0l93t/_remote_module_non_scriptable.py 2022-09-27T15:54:23.0213350Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:23.0222759Z 2022-09-27T15:54:23.0222973Z Running tests... 2022-09-27T15:54:23.0223318Z ---------------------------------------------------------------------- 2022-09-27T15:54:23.2654707Z test_rref_type_with_error_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8090 2022-09-27T15:54:23.2683453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8091 2022-09-27T15:54:23.2711418Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8092 2022-09-27T15:54:23.2740798Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8093 2022-09-27T15:54:24.3662155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:24.3662821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:24.3744348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:24.3744993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:24.4004341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:24.4005152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:24.4032964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:24.4033658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:24.4116290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:24.4116921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:24.4138473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:24.4139096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:24.4376216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:24.4376884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:24.4497520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:24.4498163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:24.5306440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdcouaxpq 2022-09-27T15:54:24.5307262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdcouaxpq/_remote_module_non_scriptable.py 2022-09-27T15:54:24.5388417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegdxl9lv 2022-09-27T15:54:24.5389697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegdxl9lv/_remote_module_non_scriptable.py 2022-09-27T15:54:24.5799985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplleu5vxo 2022-09-27T15:54:24.5800732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplleu5vxo/_remote_module_non_scriptable.py 2022-09-27T15:54:24.5812732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiird1sdu 2022-09-27T15:54:24.5815472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiird1sdu/_remote_module_non_scriptable.py 2022-09-27T15:54:24.7882666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:24.7936807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:24.8353958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:54:24.8357083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:25.0468318Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:54:25.0469061Z ValueError('Expected error') 2022-09-27T15:54:25.0469544Z Traceback (most recent call last): 2022-09-27T15:54:25.0470426Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:25.0471231Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:25.0472206Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:25.0472836Z raise ValueError(expected_err) 2022-09-27T15:54:25.0483046Z ValueError: Expected error 2022-09-27T15:54:25.0483327Z 2022-09-27T15:54:25.0483535Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:54:25.0486123Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-09-27T15:54:25.0487632Z Traceback (most recent call last): 2022-09-27T15:54:25.0488473Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:25.0489201Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:25.0490116Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-09-27T15:54:25.0490731Z rref_type = type(rref.local_value()) 2022-09-27T15:54:25.0491589Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-09-27T15:54:25.0492521Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-09-27T15:54:25.0493134Z ValueError: On WorkerInfo(id=1, name=worker1): 2022-09-27T15:54:25.0493665Z ValueError('Expected error') 2022-09-27T15:54:25.0494119Z Traceback (most recent call last): 2022-09-27T15:54:25.0494951Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:25.0495637Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:25.0496601Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:25.0497255Z raise ValueError(expected_err) 2022-09-27T15:54:25.0497672Z ValueError: Expected error 2022-09-27T15:54:25.0497931Z 2022-09-27T15:54:25.0497940Z 2022-09-27T15:54:25.0624186Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:54:25.0625044Z ValueError('Expected error') 2022-09-27T15:54:25.0625667Z Traceback (most recent call last): 2022-09-27T15:54:25.0626787Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:25.0627547Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:25.0628509Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:25.0629147Z raise ValueError(expected_err) 2022-09-27T15:54:25.0629586Z ValueError: Expected error 2022-09-27T15:54:25.0629845Z 2022-09-27T15:54:25.0637446Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:54:25.0639896Z ValueError('On WorkerInfo(id=2, name=worker2):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-09-27T15:54:25.0641451Z Traceback (most recent call last): 2022-09-27T15:54:25.0642361Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:25.0643115Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:25.0644046Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-09-27T15:54:25.0644686Z rref_type = type(rref.local_value()) 2022-09-27T15:54:25.0645557Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-09-27T15:54:25.0646453Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-09-27T15:54:25.0647069Z ValueError: On WorkerInfo(id=2, name=worker2): 2022-09-27T15:54:25.0647606Z ValueError('Expected error') 2022-09-27T15:54:25.0648037Z Traceback (most recent call last): 2022-09-27T15:54:25.0648860Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:25.0649882Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:25.0650834Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:25.0651563Z raise ValueError(expected_err) 2022-09-27T15:54:25.0651999Z ValueError: Expected error 2022-09-27T15:54:25.0652259Z 2022-09-27T15:54:25.0652270Z 2022-09-27T15:54:25.0665152Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:54:25.0665669Z ValueError('Expected error') 2022-09-27T15:54:25.0666050Z Traceback (most recent call last): 2022-09-27T15:54:25.0666740Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:25.0668640Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:25.0670161Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:25.0672662Z raise ValueError(expected_err) 2022-09-27T15:54:25.0673421Z ValueError: Expected error 2022-09-27T15:54:25.0673802Z 2022-09-27T15:54:25.0674126Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:54:25.0681798Z ValueError('On WorkerInfo(id=0, name=worker0):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-09-27T15:54:25.0683377Z Traceback (most recent call last): 2022-09-27T15:54:25.0684356Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:25.0684950Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:25.0685694Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-09-27T15:54:25.0686235Z rref_type = type(rref.local_value()) 2022-09-27T15:54:25.0689729Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-09-27T15:54:25.0690714Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-09-27T15:54:25.0693436Z ValueError: On WorkerInfo(id=0, name=worker0): 2022-09-27T15:54:25.0694063Z ValueError('Expected error') 2022-09-27T15:54:25.0696958Z Traceback (most recent call last): 2022-09-27T15:54:25.0697781Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:25.0700864Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:25.0701858Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:25.0702513Z raise ValueError(expected_err) 2022-09-27T15:54:25.0705490Z ValueError: Expected error 2022-09-27T15:54:25.0705880Z 2022-09-27T15:54:25.0708015Z 2022-09-27T15:54:25.0967935Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:54:25.0968593Z ValueError('Expected error') 2022-09-27T15:54:25.0969037Z Traceback (most recent call last): 2022-09-27T15:54:25.0969890Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:25.0970603Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:25.0971545Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:25.0972204Z raise ValueError(expected_err) 2022-09-27T15:54:25.0972652Z ValueError: Expected error 2022-09-27T15:54:25.0972911Z 2022-09-27T15:54:25.0973171Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:54:25.0975870Z ValueError('On WorkerInfo(id=3, name=worker3):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-09-27T15:54:25.0977450Z Traceback (most recent call last): 2022-09-27T15:54:25.0978264Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:25.0978976Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:25.0979881Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-09-27T15:54:25.0980510Z rref_type = type(rref.local_value()) 2022-09-27T15:54:25.0981351Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-09-27T15:54:25.0982274Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-09-27T15:54:25.0982880Z ValueError: On WorkerInfo(id=3, name=worker3): 2022-09-27T15:54:25.0983398Z ValueError('Expected error') 2022-09-27T15:54:25.0983848Z Traceback (most recent call last): 2022-09-27T15:54:25.0984672Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:25.0985359Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:25.0986313Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:25.0986954Z raise ValueError(expected_err) 2022-09-27T15:54:25.0987387Z ValueError: Expected error 2022-09-27T15:54:25.0987624Z 2022-09-27T15:54:25.0987634Z 2022-09-27T15:54:25.4798774Z ok (2.457s) 2022-09-27T15:54:25.4799051Z 2022-09-27T15:54:25.4799490Z ---------------------------------------------------------------------- 2022-09-27T15:54:25.4799769Z Ran 1 test in 2.457s 2022-09-27T15:54:25.4799873Z 2022-09-27T15:54:25.4799933Z OK 2022-09-27T15:54:25.4800027Z 2022-09-27T15:54:25.4800125Z Generating XML reports... 2022-09-27T15:54:25.4833979Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155423.xml 2022-09-27T15:54:26.8656821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:26.8657248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:26.8664931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:26.8665310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:27.0242257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbb8lwe79 2022-09-27T15:54:27.0243496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbb8lwe79/_remote_module_non_scriptable.py 2022-09-27T15:54:27.2878212Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:27.2887791Z 2022-09-27T15:54:27.2887917Z Running tests... 2022-09-27T15:54:27.2888511Z ---------------------------------------------------------------------- 2022-09-27T15:54:27.5349384Z test_rref_type_with_error_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8313 2022-09-27T15:54:27.5376633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8314 2022-09-27T15:54:27.5403809Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8315 2022-09-27T15:54:27.5432762Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8316 2022-09-27T15:54:28.6424309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:28.6425309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:28.6613947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:28.6614558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:28.6668952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:28.6669588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:28.6780816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:28.6781523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:28.6795865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:28.6796529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:28.6993652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:28.6994304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:28.7045354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:28.7046022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:28.7149735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:28.7150378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:28.8069322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuu9r7qs_ 2022-09-27T15:54:28.8070073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuu9r7qs_/_remote_module_non_scriptable.py 2022-09-27T15:54:28.8256154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpowwp5xey 2022-09-27T15:54:28.8257096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpowwp5xey/_remote_module_non_scriptable.py 2022-09-27T15:54:28.8309043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsfnlc1z 2022-09-27T15:54:28.8310466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsfnlc1z/_remote_module_non_scriptable.py 2022-09-27T15:54:28.8411269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpon8t3324 2022-09-27T15:54:28.8412442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpon8t3324/_remote_module_non_scriptable.py 2022-09-27T15:54:29.0640442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:29.0812375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:54:29.0847053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:29.0963087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:29.2745580Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:54:29.2746340Z ValueError('Expected error') 2022-09-27T15:54:29.2746800Z Traceback (most recent call last): 2022-09-27T15:54:29.2747657Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:29.2748390Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:29.2749380Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:29.2750031Z raise ValueError(expected_err) 2022-09-27T15:54:29.2750789Z ValueError: Expected error 2022-09-27T15:54:29.2751053Z 2022-09-27T15:54:29.2757573Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:54:29.2760044Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-09-27T15:54:29.2761682Z Traceback (most recent call last): 2022-09-27T15:54:29.2762907Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:29.2764265Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:29.2765647Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-09-27T15:54:29.2766475Z rref_type = type(rref.local_value()) 2022-09-27T15:54:29.2767547Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-09-27T15:54:29.2768763Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-09-27T15:54:29.2769551Z ValueError: On WorkerInfo(id=1, name=worker1): 2022-09-27T15:54:29.2770262Z ValueError('Expected error') 2022-09-27T15:54:29.2770730Z Traceback (most recent call last): 2022-09-27T15:54:29.2771859Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:29.2772718Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:29.2773852Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:29.2774800Z raise ValueError(expected_err) 2022-09-27T15:54:29.2775257Z ValueError: Expected error 2022-09-27T15:54:29.2775676Z 2022-09-27T15:54:29.2775694Z 2022-09-27T15:54:29.2948019Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:54:29.2948583Z ValueError('Expected error') 2022-09-27T15:54:29.2948954Z Traceback (most recent call last): 2022-09-27T15:54:29.2949663Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:29.2952320Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:29.2953953Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:29.2955771Z raise ValueError(expected_err) 2022-09-27T15:54:29.2956388Z ValueError: Expected error 2022-09-27T15:54:29.2956758Z 2022-09-27T15:54:29.2957048Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:54:29.2958830Z ValueError('On WorkerInfo(id=0, name=worker0):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-09-27T15:54:29.2959960Z Traceback (most recent call last): 2022-09-27T15:54:29.2960621Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:29.2961171Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:29.2961890Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-09-27T15:54:29.2962397Z rref_type = type(rref.local_value()) 2022-09-27T15:54:29.2963362Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-09-27T15:54:29.2964078Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-09-27T15:54:29.2964659Z ValueError: On WorkerInfo(id=0, name=worker0): 2022-09-27T15:54:29.2965101Z ValueError('Expected error') 2022-09-27T15:54:29.2965449Z Traceback (most recent call last): 2022-09-27T15:54:29.2966103Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:29.2966659Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:29.2967405Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:29.2967903Z raise ValueError(expected_err) 2022-09-27T15:54:29.2968265Z ValueError: Expected error 2022-09-27T15:54:29.2968477Z 2022-09-27T15:54:29.2968483Z 2022-09-27T15:54:29.3034246Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:54:29.3035059Z ValueError('Expected error') 2022-09-27T15:54:29.3035512Z Traceback (most recent call last): 2022-09-27T15:54:29.3037632Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:29.3038381Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:29.3040671Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:29.3044705Z raise ValueError(expected_err) 2022-09-27T15:54:29.3045165Z ValueError: Expected error 2022-09-27T15:54:29.3045430Z 2022-09-27T15:54:29.3045643Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:54:29.3047849Z ValueError('On WorkerInfo(id=3, name=worker3):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-09-27T15:54:29.3049270Z Traceback (most recent call last): 2022-09-27T15:54:29.3050098Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:29.3050797Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:29.3051669Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-09-27T15:54:29.3052294Z rref_type = type(rref.local_value()) 2022-09-27T15:54:29.3053143Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-09-27T15:54:29.3054065Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-09-27T15:54:29.3054661Z ValueError: On WorkerInfo(id=3, name=worker3): 2022-09-27T15:54:29.3055200Z ValueError('Expected error') 2022-09-27T15:54:29.3055641Z Traceback (most recent call last): 2022-09-27T15:54:29.3056458Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:29.3057158Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:29.3058118Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:29.3058735Z raise ValueError(expected_err) 2022-09-27T15:54:29.3059173Z ValueError: Expected error 2022-09-27T15:54:29.3059433Z 2022-09-27T15:54:29.3059442Z 2022-09-27T15:54:29.3089282Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:54:29.3089912Z ValueError('Expected error') 2022-09-27T15:54:29.3090348Z Traceback (most recent call last): 2022-09-27T15:54:29.3091533Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:29.3092248Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:29.3093305Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:29.3093964Z raise ValueError(expected_err) 2022-09-27T15:54:29.3094411Z ValueError: Expected error 2022-09-27T15:54:29.3094669Z 2022-09-27T15:54:29.3104197Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:54:29.3107499Z ValueError('On WorkerInfo(id=2, name=worker2):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-09-27T15:54:29.3108744Z Traceback (most recent call last): 2022-09-27T15:54:29.3109463Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:29.3110030Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:29.3110760Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner 2022-09-27T15:54:29.3111353Z rref_type = type(rref.local_value()) 2022-09-27T15:54:29.3112022Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-09-27T15:54:29.3112735Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-09-27T15:54:29.3113234Z ValueError: On WorkerInfo(id=2, name=worker2): 2022-09-27T15:54:29.3113670Z ValueError('Expected error') 2022-09-27T15:54:29.3114017Z Traceback (most recent call last): 2022-09-27T15:54:29.3114857Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:54:29.3115416Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:54:29.3116151Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:54:29.3116669Z raise ValueError(expected_err) 2022-09-27T15:54:29.3117038Z ValueError: Expected error 2022-09-27T15:54:29.3117250Z 2022-09-27T15:54:29.3117256Z 2022-09-27T15:54:29.7489922Z ok (2.460s) 2022-09-27T15:54:29.7490186Z 2022-09-27T15:54:29.7490710Z ---------------------------------------------------------------------- 2022-09-27T15:54:29.7490988Z Ran 1 test in 2.460s 2022-09-27T15:54:29.7491106Z 2022-09-27T15:54:29.7491166Z OK 2022-09-27T15:54:29.7491262Z 2022-09-27T15:54:29.7491359Z Generating XML reports... 2022-09-27T15:54:29.7525913Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155427.xml 2022-09-27T15:54:31.1168750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:31.1169373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:31.1177581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:31.1178215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:31.2741239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2u14wboh 2022-09-27T15:54:31.2742147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2u14wboh/_remote_module_non_scriptable.py 2022-09-27T15:54:31.5367776Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:31.5377566Z 2022-09-27T15:54:31.5377885Z Running tests... 2022-09-27T15:54:31.5378789Z ---------------------------------------------------------------------- 2022-09-27T15:54:31.7815885Z test_scalar_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8536 2022-09-27T15:54:31.7842628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8537 2022-09-27T15:54:31.7870140Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8538 2022-09-27T15:54:31.7899164Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8539 2022-09-27T15:54:32.9037224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:32.9037906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:32.9404561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:32.9405149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:32.9566260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:32.9566912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:32.9755072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:32.9755682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:32.9833237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:32.9833847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:32.9968938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:32.9969590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:33.0168486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:33.0169174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:33.0246373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:33.0247029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:33.0703771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitsaijd0 2022-09-27T15:54:33.0704568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitsaijd0/_remote_module_non_scriptable.py 2022-09-27T15:54:33.1259402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvuxrpoo 2022-09-27T15:54:33.1260192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvuxrpoo/_remote_module_non_scriptable.py 2022-09-27T15:54:33.1463533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpesvasowa 2022-09-27T15:54:33.1464800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpesvasowa/_remote_module_non_scriptable.py 2022-09-27T15:54:33.1543615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq17mp1ak 2022-09-27T15:54:33.1545397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq17mp1ak/_remote_module_non_scriptable.py 2022-09-27T15:54:33.3300704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:54:33.3844828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:33.4066541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:33.4157748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:34.0957889Z ok (2.558s) 2022-09-27T15:54:34.0959330Z 2022-09-27T15:54:34.0960309Z ---------------------------------------------------------------------- 2022-09-27T15:54:34.0960812Z Ran 1 test in 2.558s 2022-09-27T15:54:34.0960952Z 2022-09-27T15:54:34.0961015Z OK 2022-09-27T15:54:34.0961173Z 2022-09-27T15:54:34.0961267Z Generating XML reports... 2022-09-27T15:54:34.0994657Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155431.xml 2022-09-27T15:54:35.4824763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:35.4825136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:35.4833080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:35.4833506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:35.6421163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lh2abtd 2022-09-27T15:54:35.6422085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lh2abtd/_remote_module_non_scriptable.py 2022-09-27T15:54:35.9040478Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:35.9050232Z 2022-09-27T15:54:35.9050452Z Running tests... 2022-09-27T15:54:35.9050952Z ---------------------------------------------------------------------- 2022-09-27T15:54:36.1494069Z test_self_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8759 2022-09-27T15:54:36.1521889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8760 2022-09-27T15:54:36.1549383Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8761 2022-09-27T15:54:36.1579456Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8762 2022-09-27T15:54:37.3296857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:37.3297515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:37.3461619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:37.3462250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:37.3497369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:37.3498051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:37.3682643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:37.3683204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:37.3849827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:37.3850574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:37.3882513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:37.3883140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:37.3894027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:37.3894611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:37.4227759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:37.4228450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:37.4943596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmsdc7cew 2022-09-27T15:54:37.4944400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmsdc7cew/_remote_module_non_scriptable.py 2022-09-27T15:54:37.5118132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmy9hmbhm 2022-09-27T15:54:37.5119569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmy9hmbhm/_remote_module_non_scriptable.py 2022-09-27T15:54:37.5144201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodty_guv 2022-09-27T15:54:37.5145592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodty_guv/_remote_module_non_scriptable.py 2022-09-27T15:54:37.5475652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytpa_gdv 2022-09-27T15:54:37.5476454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytpa_gdv/_remote_module_non_scriptable.py 2022-09-27T15:54:37.7499485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:37.7685896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:37.7727064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:37.8038089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:54:38.4638556Z ok (2.558s) 2022-09-27T15:54:38.4638757Z 2022-09-27T15:54:38.4639090Z ---------------------------------------------------------------------- 2022-09-27T15:54:38.4639519Z Ran 1 test in 2.559s 2022-09-27T15:54:38.4639727Z 2022-09-27T15:54:38.4639823Z OK 2022-09-27T15:54:38.4639977Z 2022-09-27T15:54:38.4640074Z Generating XML reports... 2022-09-27T15:54:38.4673924Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155435.xml 2022-09-27T15:54:39.8499339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:39.8499714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:39.8507466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:39.8507939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:40.0104992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2bxinbdu 2022-09-27T15:54:40.0106066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2bxinbdu/_remote_module_non_scriptable.py 2022-09-27T15:54:40.2757647Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:40.2767267Z 2022-09-27T15:54:40.2767359Z Running tests... 2022-09-27T15:54:40.2768182Z ---------------------------------------------------------------------- 2022-09-27T15:54:40.5206139Z test_self_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8982 2022-09-27T15:54:40.5232781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8983 2022-09-27T15:54:40.5261181Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8984 2022-09-27T15:54:40.5290700Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8985 2022-09-27T15:54:41.6164963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:41.6165539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:41.6336148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:41.6336881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:41.6542170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:41.6542822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:41.6555129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:41.6555790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:41.6562690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:41.6563302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:41.6720361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:41.6720924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:41.6932270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:41.6932862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:41.6941733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:41.6942276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:41.7822424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo28a9xuk 2022-09-27T15:54:41.7823380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo28a9xuk/_remote_module_non_scriptable.py 2022-09-27T15:54:41.7989014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx7__s1ye 2022-09-27T15:54:41.7990188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx7__s1ye/_remote_module_non_scriptable.py 2022-09-27T15:54:41.8197834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxcj426s 2022-09-27T15:54:41.8198642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5qx6lkd7 2022-09-27T15:54:41.8199609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxcj426s/_remote_module_non_scriptable.py 2022-09-27T15:54:41.8200391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5qx6lkd7/_remote_module_non_scriptable.py 2022-09-27T15:54:42.0485257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:42.0632517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:54:42.0835934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:42.0938847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:42.7347015Z ok (2.458s) 2022-09-27T15:54:42.7347222Z 2022-09-27T15:54:42.7347721Z ---------------------------------------------------------------------- 2022-09-27T15:54:42.7348181Z Ran 1 test in 2.458s 2022-09-27T15:54:42.7348311Z 2022-09-27T15:54:42.7348372Z OK 2022-09-27T15:54:42.7348451Z 2022-09-27T15:54:42.7348542Z Generating XML reports... 2022-09-27T15:54:42.7385536Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155440.xml 2022-09-27T15:54:44.1486150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:44.1486817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:44.1495861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:44.1496497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:44.3064866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ug_edd5 2022-09-27T15:54:44.3065533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ug_edd5/_remote_module_non_scriptable.py 2022-09-27T15:54:44.5681997Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:44.5691872Z 2022-09-27T15:54:44.5692327Z Running tests... 2022-09-27T15:54:44.5692965Z ---------------------------------------------------------------------- 2022-09-27T15:54:44.8131745Z test_self_remote_rref_as_remote_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9205 2022-09-27T15:54:44.8159387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9206 2022-09-27T15:54:44.8187072Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9207 2022-09-27T15:54:44.8217085Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9208 2022-09-27T15:54:45.9746181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:45.9746805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:45.9907085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:45.9907785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:46.0022446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:46.0022980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:46.0121097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:46.0121641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:46.0282570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:46.0283267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:46.0394154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:46.0394965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:46.0439353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:46.0440029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:46.0809699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:46.0810287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:46.1386909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_jnw3i9 2022-09-27T15:54:46.1387725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_jnw3i9/_remote_module_non_scriptable.py 2022-09-27T15:54:46.1550531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbwquzvzt 2022-09-27T15:54:46.1551881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbwquzvzt/_remote_module_non_scriptable.py 2022-09-27T15:54:46.1657034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzslq78k 2022-09-27T15:54:46.1657971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzslq78k/_remote_module_non_scriptable.py 2022-09-27T15:54:46.2070153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1st2hmi 2022-09-27T15:54:46.2071709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1st2hmi/_remote_module_non_scriptable.py 2022-09-27T15:54:46.3979144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:46.4211778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:46.4287189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:54:46.4664106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:47.2277599Z ok (2.658s) 2022-09-27T15:54:47.2278125Z 2022-09-27T15:54:47.2278439Z ---------------------------------------------------------------------- 2022-09-27T15:54:47.2279048Z Ran 1 test in 2.659s 2022-09-27T15:54:47.2279267Z 2022-09-27T15:54:47.2279355Z OK 2022-09-27T15:54:47.2279524Z 2022-09-27T15:54:47.2279692Z Generating XML reports... 2022-09-27T15:54:47.2314748Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155444.xml 2022-09-27T15:54:48.6179101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:48.6179535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:48.6187305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:48.6187652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:48.7777216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpowtyrhoa 2022-09-27T15:54:48.7778468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpowtyrhoa/_remote_module_non_scriptable.py 2022-09-27T15:54:49.0392277Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:49.0401993Z 2022-09-27T15:54:49.0402279Z Running tests... 2022-09-27T15:54:49.0402886Z ---------------------------------------------------------------------- 2022-09-27T15:54:49.2817415Z test_self_remote_rref_as_rpc_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9428 2022-09-27T15:54:49.2845762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9429 2022-09-27T15:54:49.2873980Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9430 2022-09-27T15:54:49.2911869Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9431 2022-09-27T15:54:50.4037478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:50.4038178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:50.4118377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:50.4118906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:50.4159146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:50.4159614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:50.4428904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:50.4429542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:50.4506435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:50.4507024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:50.4521298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:50.4521878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:50.4551034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:50.4551711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:50.4901421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:50.4902069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:50.5711129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpismp969b 2022-09-27T15:54:50.5712123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpismp969b/_remote_module_non_scriptable.py 2022-09-27T15:54:50.5779041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc172_yzp 2022-09-27T15:54:50.5780302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc172_yzp/_remote_module_non_scriptable.py 2022-09-27T15:54:50.5820988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoauw_0vs 2022-09-27T15:54:50.5822089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoauw_0vs/_remote_module_non_scriptable.py 2022-09-27T15:54:50.6163772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxrrootr 2022-09-27T15:54:50.6164587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxrrootr/_remote_module_non_scriptable.py 2022-09-27T15:54:50.8261700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:50.8318530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:50.8366537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:50.8715882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:54:51.5968717Z ok (2.556s) 2022-09-27T15:54:51.5968983Z 2022-09-27T15:54:51.5969325Z ---------------------------------------------------------------------- 2022-09-27T15:54:51.5969576Z Ran 1 test in 2.557s 2022-09-27T15:54:51.5969690Z 2022-09-27T15:54:51.5969754Z OK 2022-09-27T15:54:51.5969883Z 2022-09-27T15:54:51.5969979Z Generating XML reports... 2022-09-27T15:54:51.6005321Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155449.xml 2022-09-27T15:54:53.0007518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:53.0007918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:53.0015502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:53.0015867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:53.1595563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpooz0uwnj 2022-09-27T15:54:53.1596474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpooz0uwnj/_remote_module_non_scriptable.py 2022-09-27T15:54:53.4197593Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:53.4207500Z 2022-09-27T15:54:53.4207603Z Running tests... 2022-09-27T15:54:53.4208669Z ---------------------------------------------------------------------- 2022-09-27T15:54:53.6675206Z test_self_remote_rref_as_self_remote_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9651 2022-09-27T15:54:53.6702791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9652 2022-09-27T15:54:53.6731924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9653 2022-09-27T15:54:53.6761836Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9654 2022-09-27T15:54:54.8206658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:54.8207311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:54.8578140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:54.8578801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:54.8836572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:54.8837455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:54.8902475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:54.8903294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:54.9003952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:54.9004477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:54.9208301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:54.9208961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:54.9279821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:54.9280509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:54.9373335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:54.9374022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:54.9853851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89imapgl 2022-09-27T15:54:54.9854610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89imapgl/_remote_module_non_scriptable.py 2022-09-27T15:54:55.0477816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelrdkka_ 2022-09-27T15:54:55.0478612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelrdkka_/_remote_module_non_scriptable.py 2022-09-27T15:54:55.0563897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvt_vw1u 2022-09-27T15:54:55.0564626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvt_vw1u/_remote_module_non_scriptable.py 2022-09-27T15:54:55.0644340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2m0oma6h 2022-09-27T15:54:55.0645087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2m0oma6h/_remote_module_non_scriptable.py 2022-09-27T15:54:55.2457876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:55.3097064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:55.3198843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:55.3273737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:54:55.9819328Z ok (2.561s) 2022-09-27T15:54:55.9819485Z 2022-09-27T15:54:55.9819873Z ---------------------------------------------------------------------- 2022-09-27T15:54:55.9820125Z Ran 1 test in 2.561s 2022-09-27T15:54:55.9820244Z 2022-09-27T15:54:55.9820295Z OK 2022-09-27T15:54:55.9820407Z 2022-09-27T15:54:55.9820505Z Generating XML reports... 2022-09-27T15:54:55.9856127Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155453.xml 2022-09-27T15:54:57.3716338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:57.3724874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:57.3725690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:57.3726288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:57.5292003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprgne5fph 2022-09-27T15:54:57.5293035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprgne5fph/_remote_module_non_scriptable.py 2022-09-27T15:54:57.7896479Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:54:57.7906225Z 2022-09-27T15:54:57.7906544Z Running tests... 2022-09-27T15:54:57.7907344Z ---------------------------------------------------------------------- 2022-09-27T15:54:58.0321926Z test_self_remote_rref_as_self_rpc_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9874 2022-09-27T15:54:58.0348843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9875 2022-09-27T15:54:58.0376845Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9876 2022-09-27T15:54:58.0406161Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9877 2022-09-27T15:54:59.1786168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:59.1786833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:59.1985662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:59.1986312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:59.2158302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:59.2158927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:59.2367934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:59.2368563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:59.2470092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:59.2470745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:59.2767802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:54:59.2768443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:54:59.2844620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:59.2845290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:59.3144458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:54:59.3145109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:54:59.3438627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyl105xa_ 2022-09-27T15:54:59.3439649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyl105xa_/_remote_module_non_scriptable.py 2022-09-27T15:54:59.3640042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9i9drqzk 2022-09-27T15:54:59.3641334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9i9drqzk/_remote_module_non_scriptable.py 2022-09-27T15:54:59.4117736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgldf2r4a 2022-09-27T15:54:59.4118456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgldf2r4a/_remote_module_non_scriptable.py 2022-09-27T15:54:59.4419206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkx27slbk 2022-09-27T15:54:59.4419952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkx27slbk/_remote_module_non_scriptable.py 2022-09-27T15:54:59.6026738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:54:59.6217840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:54:59.6728864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:54:59.7005134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:00.3464654Z ok (2.555s) 2022-09-27T15:55:00.3465282Z 2022-09-27T15:55:00.3465745Z ---------------------------------------------------------------------- 2022-09-27T15:55:00.3465989Z Ran 1 test in 2.556s 2022-09-27T15:55:00.3466105Z 2022-09-27T15:55:00.3466164Z OK 2022-09-27T15:55:00.3466255Z 2022-09-27T15:55:00.3466349Z Generating XML reports... 2022-09-27T15:55:00.3500646Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155457.xml 2022-09-27T15:55:01.7232010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:01.7232455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:01.7240913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:01.7241298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:01.8845206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgo3inrfm 2022-09-27T15:55:01.8846705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgo3inrfm/_remote_module_non_scriptable.py 2022-09-27T15:55:02.1531782Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:02.1541852Z 2022-09-27T15:55:02.1542121Z Running tests... 2022-09-27T15:55:02.1542570Z ---------------------------------------------------------------------- 2022-09-27T15:55:02.4024086Z test_send_to_rank (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10097 2022-09-27T15:55:02.4051546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10098 2022-09-27T15:55:02.4078828Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10099 2022-09-27T15:55:02.4108269Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10100 2022-09-27T15:55:03.5272408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:03.5273099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:03.5323617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:03.5324195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:03.5375429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:03.5376097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:03.5417851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:03.5418524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:03.5653258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:03.5653888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:03.5703785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:03.5704419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:03.5755281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:03.5755900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:03.5797190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:03.5798072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:03.6916529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5t4mni77 2022-09-27T15:55:03.6917880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5t4mni77/_remote_module_non_scriptable.py 2022-09-27T15:55:03.6961412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0ckfw35 2022-09-27T15:55:03.6963490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0ckfw35/_remote_module_non_scriptable.py 2022-09-27T15:55:03.7006763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fglt0dl 2022-09-27T15:55:03.7008302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fglt0dl/_remote_module_non_scriptable.py 2022-09-27T15:55:03.7058101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44aytyyu 2022-09-27T15:55:03.7059979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44aytyyu/_remote_module_non_scriptable.py 2022-09-27T15:55:03.9459423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:03.9585237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:55:03.9648249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:03.9715507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:04.6164813Z ok (2.462s) 2022-09-27T15:55:04.6165098Z 2022-09-27T15:55:04.6165602Z ---------------------------------------------------------------------- 2022-09-27T15:55:04.6165983Z Ran 1 test in 2.462s 2022-09-27T15:55:04.6166102Z 2022-09-27T15:55:04.6166162Z OK 2022-09-27T15:55:04.6166253Z 2022-09-27T15:55:04.6166335Z Generating XML reports... 2022-09-27T15:55:04.6203610Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155502.xml 2022-09-27T15:55:05.9880850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:05.9881238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:05.9888660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:05.9889191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:06.1466899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5joev38q 2022-09-27T15:55:06.1467848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5joev38q/_remote_module_non_scriptable.py 2022-09-27T15:55:06.4097059Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:06.4106957Z 2022-09-27T15:55:06.4107126Z Running tests... 2022-09-27T15:55:06.4107529Z ---------------------------------------------------------------------- 2022-09-27T15:55:06.6497400Z test_server_process_global_profiler (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10320 2022-09-27T15:55:06.6525117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10321 2022-09-27T15:55:06.6553824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10322 2022-09-27T15:55:06.6583736Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10323 2022-09-27T15:55:07.7490236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:07.7490933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:07.7863667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:07.7864317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:07.7870592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:07.7871728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:07.7872980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:07.7873523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:07.7875485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:07.7876066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:07.8262188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:07.8262848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:07.8265103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:07.8265843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:07.8266814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:07.8267392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:07.9148225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpldnr46vq 2022-09-27T15:55:07.9149560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpldnr46vq/_remote_module_non_scriptable.py 2022-09-27T15:55:07.9551048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgfzhk83m 2022-09-27T15:55:07.9552121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgfzhk83m/_remote_module_non_scriptable.py 2022-09-27T15:55:07.9559671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57266fbp 2022-09-27T15:55:07.9561196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57266fbp/_remote_module_non_scriptable.py 2022-09-27T15:55:07.9749929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjlilpwv 2022-09-27T15:55:07.9752097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjlilpwv/_remote_module_non_scriptable.py 2022-09-27T15:55:08.1949635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:08.2144998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:08.2175015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:55:08.2472139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:08.8640730Z ok (2.453s) 2022-09-27T15:55:08.8640928Z 2022-09-27T15:55:08.8641228Z ---------------------------------------------------------------------- 2022-09-27T15:55:08.8641481Z Ran 1 test in 2.453s 2022-09-27T15:55:08.8641594Z 2022-09-27T15:55:08.8641653Z OK 2022-09-27T15:55:08.8641754Z 2022-09-27T15:55:08.8641847Z Generating XML reports... 2022-09-27T15:55:08.8676734Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155506.xml 2022-09-27T15:55:10.2124744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:10.2125367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:10.2133817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:10.2134436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:10.3712253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptibuwwk0 2022-09-27T15:55:10.3713400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptibuwwk0/_remote_module_non_scriptable.py 2022-09-27T15:55:10.6308412Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:10.6318604Z 2022-09-27T15:55:10.6318900Z Running tests... 2022-09-27T15:55:10.6319520Z ---------------------------------------------------------------------- 2022-09-27T15:55:10.8749265Z test_set_and_get_default_rpc_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10543 2022-09-27T15:55:10.8776409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10544 2022-09-27T15:55:10.8804126Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10545 2022-09-27T15:55:10.8834949Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10546 2022-09-27T15:55:12.0153036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:12.0153717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:12.0296900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:12.0297442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:12.0525948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:12.0526521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:12.0686171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:12.0686806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:12.1187256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:12.1187899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:12.1488916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:12.1489577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:12.1545593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:12.1546248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:12.1795416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3fmzam_ 2022-09-27T15:55:12.1796051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3fmzam_/_remote_module_non_scriptable.py 2022-09-27T15:55:12.1844989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:12.1845631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:12.1967414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4ccxc3p 2022-09-27T15:55:12.1968604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4ccxc3p/_remote_module_non_scriptable.py 2022-09-27T15:55:12.2842749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpceb6xwzw 2022-09-27T15:55:12.2843711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpceb6xwzw/_remote_module_non_scriptable.py 2022-09-27T15:55:12.3117705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ql2etui 2022-09-27T15:55:12.3118836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ql2etui/_remote_module_non_scriptable.py 2022-09-27T15:55:12.4381307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:12.4555034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:12.5420035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:55:12.5676855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:13.1891480Z ok (2.557s) 2022-09-27T15:55:13.1891660Z 2022-09-27T15:55:13.1891969Z ---------------------------------------------------------------------- 2022-09-27T15:55:13.1892208Z Ran 1 test in 2.557s 2022-09-27T15:55:13.1892327Z 2022-09-27T15:55:13.1892388Z OK 2022-09-27T15:55:13.1892484Z 2022-09-27T15:55:13.1892579Z Generating XML reports... 2022-09-27T15:55:13.1928816Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155510.xml 2022-09-27T15:55:14.5647883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:14.5648518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:14.5657678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:14.5658071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:14.7249721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0sebo98 2022-09-27T15:55:14.7250485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0sebo98/_remote_module_non_scriptable.py 2022-09-27T15:55:14.9864317Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:14.9873310Z 2022-09-27T15:55:14.9873436Z Running tests... 2022-09-27T15:55:14.9873861Z ---------------------------------------------------------------------- 2022-09-27T15:55:15.2372189Z test_shutdown_errors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10766 2022-09-27T15:55:15.2398849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10767 2022-09-27T15:55:15.2426006Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10768 2022-09-27T15:55:15.2456962Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10769 2022-09-27T15:55:16.3490453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:16.3491059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:16.3531541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:16.3532217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:16.3614343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:16.3614969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:16.3842470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:16.3853775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:16.3883697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:16.3884379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:16.3932074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:16.3932624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:16.4003320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:16.4003943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:16.4214383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:16.4215116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:16.5147036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp86txz176 2022-09-27T15:55:16.5147859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp86txz176/_remote_module_non_scriptable.py 2022-09-27T15:55:16.5209395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmkszlvw 2022-09-27T15:55:16.5211948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmkszlvw/_remote_module_non_scriptable.py 2022-09-27T15:55:16.5298546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_dyfdaq 2022-09-27T15:55:16.5300866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_dyfdaq/_remote_module_non_scriptable.py 2022-09-27T15:55:16.5468121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yueg_6r 2022-09-27T15:55:16.5469770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yueg_6r/_remote_module_non_scriptable.py 2022-09-27T15:55:16.7709587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:16.7888453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:16.7928476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:55:16.8039548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:16.8366034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:55:16.8366644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:55:16.8468208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:55:16.8469011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:55:16.8469676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:55:16.8470440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:55:16.8471007Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:55:16.8471607Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:55:17.0636651Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:55:17.0637861Z RuntimeError('simulation') 2022-09-27T15:55:17.0638632Z Traceback (most recent call last): 2022-09-27T15:55:17.0640173Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:55:17.0640930Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:55:17.0641706Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4523, in raise_error 2022-09-27T15:55:17.0642334Z raise RuntimeError('simulation') 2022-09-27T15:55:17.0642645Z RuntimeError: simulation 2022-09-27T15:55:17.0643310Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:55:17.0645008Z 2022-09-27T15:55:17.0645642Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:55:17.0646441Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:55:17.0646950Z RuntimeError('simulation') 2022-09-27T15:55:17.0647725Z Traceback (most recent call last): 2022-09-27T15:55:17.0648560Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:55:17.0649309Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:55:17.0650013Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4523, in raise_error 2022-09-27T15:55:17.0650543Z raise RuntimeError('simulation') 2022-09-27T15:55:17.0650867Z RuntimeError: simulation 2022-09-27T15:55:17.0651423Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:55:17.0651867Z 2022-09-27T15:55:17.0652361Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #4 to worker0, as the agent is shutting down 2022-09-27T15:55:17.0652808Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:55:17.0653187Z RuntimeError('simulation') 2022-09-27T15:55:17.0653482Z Traceback (most recent call last): 2022-09-27T15:55:17.0654123Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:55:17.0654664Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:55:17.0655460Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4523, in raise_error 2022-09-27T15:55:17.0655936Z raise RuntimeError('simulation') 2022-09-27T15:55:17.0656260Z RuntimeError: simulation 2022-09-27T15:55:17.0656426Z 2022-09-27T15:55:17.0656676Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:55:17.0657321Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:55:17.0657963Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:55:17.0658930Z [W tensorpipe_agent.cpp:625] RPC agent for worker2 won't send response to request #5 to worker0, as the agent is shutting down 2022-09-27T15:55:17.0659537Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:55:17.0660373Z ERROR:torch.distributed.rpc.api:Failed to respond to 'Shutdown Proceed' in time, got error Followers ['worker1', 'worker3', 'worker2'] timed out in _all_gather after 0.00 seconds. The first exception is RuntimeError: On WorkerInfo(id=1, name=worker1): 2022-09-27T15:55:17.0660858Z RuntimeError('simulation') 2022-09-27T15:55:17.0661069Z Traceback (most recent call last): 2022-09-27T15:55:17.0661552Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:55:17.0661921Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:55:17.0662390Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4523, in raise_error 2022-09-27T15:55:17.0662784Z raise RuntimeError('simulation') 2022-09-27T15:55:17.0662993Z RuntimeError: simulation 2022-09-27T15:55:17.0663111Z 2022-09-27T15:55:17.0663116Z 2022-09-27T15:55:17.0663163Z At: 2022-09-27T15:55:17.0663636Z /opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(220): _handle_exception 2022-09-27T15:55:17.0663907Z 2022-09-27T15:55:17.0664298Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:55:17.4513142Z ok (2.463s) 2022-09-27T15:55:17.4513446Z 2022-09-27T15:55:17.4513899Z ---------------------------------------------------------------------- 2022-09-27T15:55:17.4514220Z Ran 1 test in 2.464s 2022-09-27T15:55:17.4514341Z 2022-09-27T15:55:17.4514403Z OK 2022-09-27T15:55:17.4514601Z 2022-09-27T15:55:17.4514698Z Generating XML reports... 2022-09-27T15:55:17.4548941Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155514.xml 2022-09-27T15:55:18.8355453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:18.8355846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:18.8363568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:18.8363934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:18.9958287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkr3_c1uy 2022-09-27T15:55:18.9959056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkr3_c1uy/_remote_module_non_scriptable.py 2022-09-27T15:55:19.2614990Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:19.2624197Z 2022-09-27T15:55:19.2624344Z Running tests... 2022-09-27T15:55:19.2624958Z ---------------------------------------------------------------------- 2022-09-27T15:55:19.5090997Z test_shutdown_followed_by_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11001 2022-09-27T15:55:19.5117726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11002 2022-09-27T15:55:19.5145532Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11003 2022-09-27T15:55:19.5174672Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11004 2022-09-27T15:55:20.5736931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:20.5737621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:20.5784126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:20.5784786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:20.6118829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:20.6119450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:20.6126989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:20.6127652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:20.6143250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:20.6143892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:20.6161662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:20.6162340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:20.6512238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:20.6512872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:20.6531620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:20.6532281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:20.7402723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpncs7f9zz 2022-09-27T15:55:20.7403479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpncs7f9zz/_remote_module_non_scriptable.py 2022-09-27T15:55:20.7494623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafz5fscv 2022-09-27T15:55:20.7495806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafz5fscv/_remote_module_non_scriptable.py 2022-09-27T15:55:20.7782375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnb2prqnb 2022-09-27T15:55:20.7783279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnb2prqnb/_remote_module_non_scriptable.py 2022-09-27T15:55:20.7811719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7m2fh9k7 2022-09-27T15:55:20.7812547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7m2fh9k7/_remote_module_non_scriptable.py 2022-09-27T15:55:20.9999431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:21.0083407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:55:21.0370232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:21.0409373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:21.6227850Z ok (2.360s) 2022-09-27T15:55:21.6228068Z 2022-09-27T15:55:21.6228525Z ---------------------------------------------------------------------- 2022-09-27T15:55:21.6228964Z Ran 1 test in 2.360s 2022-09-27T15:55:21.6229154Z 2022-09-27T15:55:21.6229220Z OK 2022-09-27T15:55:21.6229313Z 2022-09-27T15:55:21.6229408Z Generating XML reports... 2022-09-27T15:55:21.6266848Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155519.xml 2022-09-27T15:55:22.9787646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:22.9788033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:22.9796133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:22.9796671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:23.1380724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpry6cta1u 2022-09-27T15:55:23.1381507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpry6cta1u/_remote_module_non_scriptable.py 2022-09-27T15:55:23.4007434Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:23.4017141Z 2022-09-27T15:55:23.4017413Z Running tests... 2022-09-27T15:55:23.4018000Z ---------------------------------------------------------------------- 2022-09-27T15:55:23.6468073Z test_stress_heavy_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11224 2022-09-27T15:55:23.6495655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11225 2022-09-27T15:55:23.6523475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11226 2022-09-27T15:55:23.6553297Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11227 2022-09-27T15:55:24.7986566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:24.7987260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:24.8012558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:24.8013227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:24.8152560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:24.8153344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:24.8164654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:24.8165532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:24.8368723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:24.8369274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:24.8400584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:24.8401092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:24.8536921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:24.8537549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:24.8545455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:24.8546062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:24.9649943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1r2ctp1f 2022-09-27T15:55:24.9650757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1r2ctp1f/_remote_module_non_scriptable.py 2022-09-27T15:55:24.9663265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0tom341 2022-09-27T15:55:24.9665248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0tom341/_remote_module_non_scriptable.py 2022-09-27T15:55:24.9795611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqu1ifv56 2022-09-27T15:55:24.9797119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqu1ifv56/_remote_module_non_scriptable.py 2022-09-27T15:55:24.9803229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8iv5h7bz 2022-09-27T15:55:24.9805625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8iv5h7bz/_remote_module_non_scriptable.py 2022-09-27T15:55:25.2193728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:25.2220690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:25.2346410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:55:25.2358127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:25.5577973Z Rank 0 finished testing 20 times in 0.07614660263061523 seconds. 2022-09-27T15:55:25.5696626Z Rank 3 finished testing 20 times in 0.06782674789428711 seconds. 2022-09-27T15:55:25.5816969Z Rank 2 finished testing 20 times in 0.0776975154876709 seconds. 2022-09-27T15:55:25.5924385Z Rank 1 finished testing 20 times in 0.08710384368896484 seconds. 2022-09-27T15:55:25.9612003Z ok (2.559s) 2022-09-27T15:55:25.9612226Z 2022-09-27T15:55:25.9612733Z ---------------------------------------------------------------------- 2022-09-27T15:55:25.9613091Z Ran 1 test in 2.559s 2022-09-27T15:55:25.9613211Z 2022-09-27T15:55:25.9613273Z OK 2022-09-27T15:55:25.9613367Z 2022-09-27T15:55:25.9613461Z Generating XML reports... 2022-09-27T15:55:25.9650517Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155523.xml 2022-09-27T15:55:27.3290505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:27.3290899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:27.3299497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:27.3300082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:27.4870472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6e_unloi 2022-09-27T15:55:27.4871465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6e_unloi/_remote_module_non_scriptable.py 2022-09-27T15:55:27.7602787Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:27.7612803Z 2022-09-27T15:55:27.7613253Z Running tests... 2022-09-27T15:55:27.7613685Z ---------------------------------------------------------------------- 2022-09-27T15:55:28.0061056Z test_stress_heavy_rpc_torchscript (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11447 2022-09-27T15:55:28.0089247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11448 2022-09-27T15:55:28.0117721Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11449 2022-09-27T15:55:28.0146651Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11450 2022-09-27T15:55:29.1281194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:29.1281883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:29.1315400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:29.1316041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:29.1328719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:29.1329360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:29.1575121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:29.1575771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:29.1663338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:29.1663964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:29.1703841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:29.1704436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:29.1719841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:29.1720494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:29.1942517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:29.1943193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:29.2942227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44a5no36 2022-09-27T15:55:29.2943028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44a5no36/_remote_module_non_scriptable.py 2022-09-27T15:55:29.2976797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7t4ho_e0 2022-09-27T15:55:29.2978695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7t4ho_e0/_remote_module_non_scriptable.py 2022-09-27T15:55:29.2993024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbot1qpni 2022-09-27T15:55:29.2995803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbot1qpni/_remote_module_non_scriptable.py 2022-09-27T15:55:29.3199216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi55z9k7f 2022-09-27T15:55:29.3199970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi55z9k7f/_remote_module_non_scriptable.py 2022-09-27T15:55:29.5518385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:29.5555691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:29.5583601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:55:29.5773618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:29.8980879Z Rank 0 finished testing 20 times in 0.12430024147033691 seconds. 2022-09-27T15:55:29.8987364Z Rank 3 finished testing 20 times in 0.10428690910339355 seconds. 2022-09-27T15:55:29.9101048Z Rank 2 finished testing 20 times in 0.1164541244506836 seconds. 2022-09-27T15:55:29.9197876Z Rank 1 finished testing 20 times in 0.12606287002563477 seconds. 2022-09-27T15:55:30.3206898Z ok (2.559s) 2022-09-27T15:55:30.3207186Z 2022-09-27T15:55:30.3207638Z ---------------------------------------------------------------------- 2022-09-27T15:55:30.3207906Z Ran 1 test in 2.559s 2022-09-27T15:55:30.3208024Z 2022-09-27T15:55:30.3208071Z OK 2022-09-27T15:55:30.3208174Z 2022-09-27T15:55:30.3208267Z Generating XML reports... 2022-09-27T15:55:30.3245060Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155527.xml 2022-09-27T15:55:31.7227297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:31.7227689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:31.7235798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:31.7236145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:31.8803864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6ij9q19 2022-09-27T15:55:31.8804875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6ij9q19/_remote_module_non_scriptable.py 2022-09-27T15:55:32.1433603Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:32.1443423Z 2022-09-27T15:55:32.1443707Z Running tests... 2022-09-27T15:55:32.1444293Z ---------------------------------------------------------------------- 2022-09-27T15:55:32.3852583Z test_stress_light_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11670 2022-09-27T15:55:32.3880324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11671 2022-09-27T15:55:32.3908601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11672 2022-09-27T15:55:32.3937916Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11673 2022-09-27T15:55:33.4980171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:33.4980757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:33.5363379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:33.5364085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:33.5395093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:33.5395740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:33.5400115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:33.5400750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:33.5416105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:33.5417042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:33.5793703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:33.5794827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:33.5798841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:33.5799383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:33.5814462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:33.5815100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:33.6649690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkyr80wbw 2022-09-27T15:55:33.6650767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkyr80wbw/_remote_module_non_scriptable.py 2022-09-27T15:55:33.7062616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmk0xof2o 2022-09-27T15:55:33.7063513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3t2nabn2 2022-09-27T15:55:33.7064184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmk0xof2o/_remote_module_non_scriptable.py 2022-09-27T15:55:33.7064865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsk9gscx4 2022-09-27T15:55:33.7065973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3t2nabn2/_remote_module_non_scriptable.py 2022-09-27T15:55:33.7066928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsk9gscx4/_remote_module_non_scriptable.py 2022-09-27T15:55:33.9247671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:33.9629830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:33.9648346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:33.9833342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:55:34.4450300Z Rank 0 finished testing 1000 times in 0.2435157299041748 seconds. 2022-09-27T15:55:34.4986640Z Rank 2 finished testing 1000 times in 0.27056884765625 seconds. 2022-09-27T15:55:34.5093780Z Rank 1 finished testing 1000 times in 0.28792572021484375 seconds. 2022-09-27T15:55:34.5364987Z Rank 3 finished testing 1000 times in 0.31491637229919434 seconds. 2022-09-27T15:55:34.9000568Z ok (2.755s) 2022-09-27T15:55:34.9000799Z 2022-09-27T15:55:34.9001152Z ---------------------------------------------------------------------- 2022-09-27T15:55:34.9001406Z Ran 1 test in 2.756s 2022-09-27T15:55:34.9001526Z 2022-09-27T15:55:34.9001587Z OK 2022-09-27T15:55:34.9001731Z 2022-09-27T15:55:34.9001810Z Generating XML reports... 2022-09-27T15:55:34.9039916Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155532.xml 2022-09-27T15:55:36.2744040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:36.2744414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:36.2752315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:36.2752829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:36.4331088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fp40yrh 2022-09-27T15:55:36.4331917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fp40yrh/_remote_module_non_scriptable.py 2022-09-27T15:55:36.6922480Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:36.6932074Z 2022-09-27T15:55:36.6932730Z Running tests... 2022-09-27T15:55:36.6933286Z ---------------------------------------------------------------------- 2022-09-27T15:55:36.9367903Z test_use_rpc_pickler (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11893 2022-09-27T15:55:36.9395691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11894 2022-09-27T15:55:36.9424749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11895 2022-09-27T15:55:36.9454854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11896 2022-09-27T15:55:38.1028382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:38.1029022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:38.1084980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:38.1085703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:38.1258157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:38.1258702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:38.1400184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:38.1400975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:38.1461451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:38.1462228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:38.1463058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:38.1463626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:38.1638648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:38.1639321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:38.1838231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:38.1838904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:38.2663738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv55j65y3 2022-09-27T15:55:38.2664882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv55j65y3/_remote_module_non_scriptable.py 2022-09-27T15:55:38.2724480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptzt7pa7z 2022-09-27T15:55:38.2726073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptzt7pa7z/_remote_module_non_scriptable.py 2022-09-27T15:55:38.2907406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ds68yn8 2022-09-27T15:55:38.2908554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ds68yn8/_remote_module_non_scriptable.py 2022-09-27T15:55:38.3108076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpss_y86k4 2022-09-27T15:55:38.3108727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpss_y86k4/_remote_module_non_scriptable.py 2022-09-27T15:55:38.5226215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:38.5275265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:38.5472401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:38.5909342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:55:38.9507207Z ok (2.257s) 2022-09-27T15:55:38.9507754Z 2022-09-27T15:55:38.9508258Z ---------------------------------------------------------------------- 2022-09-27T15:55:38.9508812Z Ran 1 test in 2.257s 2022-09-27T15:55:38.9509029Z 2022-09-27T15:55:38.9509133Z OK 2022-09-27T15:55:38.9509280Z 2022-09-27T15:55:38.9509439Z Generating XML reports... 2022-09-27T15:55:38.9545029Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155536.xml 2022-09-27T15:55:40.3517400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:40.3518043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:40.3526043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:40.3526425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:40.5087775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppo67kjo6 2022-09-27T15:55:40.5088393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppo67kjo6/_remote_module_non_scriptable.py 2022-09-27T15:55:40.7697892Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:40.7707703Z 2022-09-27T15:55:40.7707833Z Running tests... 2022-09-27T15:55:40.7708334Z ---------------------------------------------------------------------- 2022-09-27T15:55:41.0154107Z test_use_rref_after_shutdown (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11948 2022-09-27T15:55:41.0182073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11949 2022-09-27T15:55:41.0210209Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11950 2022-09-27T15:55:41.0240199Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11951 2022-09-27T15:55:42.1099356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:42.1100027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:42.1224848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:42.1225527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:42.1232224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:42.1232880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:42.1337435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:42.1338353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:42.1501372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:42.1501943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:42.1617907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:42.1618606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:42.1621281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:42.1621815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:42.1722809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:42.1723471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:42.2789733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy84tlnlu 2022-09-27T15:55:42.2790720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy84tlnlu/_remote_module_non_scriptable.py 2022-09-27T15:55:42.2896499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8_d3i80 2022-09-27T15:55:42.2897872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3q42xf_ 2022-09-27T15:55:42.2898824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8_d3i80/_remote_module_non_scriptable.py 2022-09-27T15:55:42.2899764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3q42xf_/_remote_module_non_scriptable.py 2022-09-27T15:55:42.2998768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjr13lrz9 2022-09-27T15:55:42.3000190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjr13lrz9/_remote_module_non_scriptable.py 2022-09-27T15:55:42.5399537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:42.5510668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:42.5526873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:42.5615037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:55:43.1293664Z ok (2.358s) 2022-09-27T15:55:43.1293902Z 2022-09-27T15:55:43.1294345Z ---------------------------------------------------------------------- 2022-09-27T15:55:43.1294629Z Ran 1 test in 2.359s 2022-09-27T15:55:43.1294747Z 2022-09-27T15:55:43.1294811Z OK 2022-09-27T15:55:43.1294948Z 2022-09-27T15:55:43.1295116Z Generating XML reports... 2022-09-27T15:55:43.1330961Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155540.xml 2022-09-27T15:55:44.5293137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:44.5293532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:44.5302161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:44.5302687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:44.6881304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wddvk4g 2022-09-27T15:55:44.6882245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wddvk4g/_remote_module_non_scriptable.py 2022-09-27T15:55:44.9510084Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:44.9518903Z 2022-09-27T15:55:44.9519046Z Running tests... 2022-09-27T15:55:44.9519637Z ---------------------------------------------------------------------- 2022-09-27T15:55:45.2025985Z test_user_rref_backward (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12171 2022-09-27T15:55:45.2054030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12172 2022-09-27T15:55:45.2082630Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12173 2022-09-27T15:55:45.2112787Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12174 2022-09-27T15:55:46.3230974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:46.3231731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:46.3342565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:46.3450021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:46.3450785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:46.3451548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:46.3577084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:46.3578088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:46.3602315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:46.3602915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:46.3718690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:46.3719338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:46.3827845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:46.3828499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:46.3932680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:46.3933415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:46.4865386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkqzpka_ 2022-09-27T15:55:46.4866651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkqzpka_/_remote_module_non_scriptable.py 2022-09-27T15:55:46.4982772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlih3hwe 2022-09-27T15:55:46.4983952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlih3hwe/_remote_module_non_scriptable.py 2022-09-27T15:55:46.5085163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7guut6n 2022-09-27T15:55:46.5085839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7guut6n/_remote_module_non_scriptable.py 2022-09-27T15:55:46.5183133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3py5why5 2022-09-27T15:55:46.5184498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3py5why5/_remote_module_non_scriptable.py 2022-09-27T15:55:46.7412537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:46.7543647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:46.7660460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:46.7721948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:55:47.4167756Z ok (2.464s) 2022-09-27T15:55:47.4167955Z 2022-09-27T15:55:47.4168389Z ---------------------------------------------------------------------- 2022-09-27T15:55:47.4168828Z Ran 1 test in 2.465s 2022-09-27T15:55:47.4169047Z 2022-09-27T15:55:47.4169146Z OK 2022-09-27T15:55:47.4169244Z 2022-09-27T15:55:47.4169356Z Generating XML reports... 2022-09-27T15:55:47.4203804Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155544.xml 2022-09-27T15:55:48.7953512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:48.7953958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:48.7962138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:48.7962576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:48.9559197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphnxwixc7 2022-09-27T15:55:48.9559734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphnxwixc7/_remote_module_non_scriptable.py 2022-09-27T15:55:49.2197897Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:49.2208067Z 2022-09-27T15:55:49.2208366Z Running tests... 2022-09-27T15:55:49.2209098Z ---------------------------------------------------------------------- 2022-09-27T15:55:49.4685380Z test_user_rrefs_confirmed (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12414 2022-09-27T15:55:49.4714341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12415 2022-09-27T15:55:49.4742642Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12416 2022-09-27T15:55:49.4773173Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12417 2022-09-27T15:55:50.6527859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:50.6528468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:50.6893643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:50.6894279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:50.6905727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:50.6906337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:50.7029685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:50.7030345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:50.7167944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:50.7168530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:50.7292684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:50.7293346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:50.7422137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:50.7422818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:50.7556491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:50.7557140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:50.8194007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwyxlilh3 2022-09-27T15:55:50.8195006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwyxlilh3/_remote_module_non_scriptable.py 2022-09-27T15:55:50.8577789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4t3_w4ky 2022-09-27T15:55:50.8578529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4t3_w4ky/_remote_module_non_scriptable.py 2022-09-27T15:55:50.8698963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfun8vz6l 2022-09-27T15:55:50.8699731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfun8vz6l/_remote_module_non_scriptable.py 2022-09-27T15:55:50.8832183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmaut_u1 2022-09-27T15:55:50.8833273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmaut_u1/_remote_module_non_scriptable.py 2022-09-27T15:55:51.0804226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:51.1200041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:51.1299478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:55:51.1417428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:51.8867748Z ok (2.666s) 2022-09-27T15:55:51.8868328Z 2022-09-27T15:55:51.8868759Z ---------------------------------------------------------------------- 2022-09-27T15:55:51.8869014Z Ran 1 test in 2.666s 2022-09-27T15:55:51.8869138Z 2022-09-27T15:55:51.8869199Z OK 2022-09-27T15:55:51.8869292Z 2022-09-27T15:55:51.8869375Z Generating XML reports... 2022-09-27T15:55:51.8906006Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155549.xml 2022-09-27T15:55:53.2597326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:53.2597705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:53.2605676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:53.2606046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:53.4194430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzenaj3gq 2022-09-27T15:55:53.4196170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzenaj3gq/_remote_module_non_scriptable.py 2022-09-27T15:55:53.6833886Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:53.6843744Z 2022-09-27T15:55:53.6843830Z Running tests... 2022-09-27T15:55:53.6844277Z ---------------------------------------------------------------------- 2022-09-27T15:55:53.9302652Z test_user_rrefs_confirmed_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12637 2022-09-27T15:55:53.9330438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12638 2022-09-27T15:55:53.9358819Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12639 2022-09-27T15:55:53.9388407Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12640 2022-09-27T15:55:55.0961456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:55.0962131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:55.1315258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:55.1315913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:55.1325927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:55.1326550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:55.1394940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:55.1395605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:55.1697759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:55.1698419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:55.1765169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:55.1765852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:55.1902511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:55.1903207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:55.2281158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:55.2281790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:55.2592216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4h8dka5e 2022-09-27T15:55:55.2593157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4h8dka5e/_remote_module_non_scriptable.py 2022-09-27T15:55:55.2974558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6oroxg7 2022-09-27T15:55:55.2975370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6oroxg7/_remote_module_non_scriptable.py 2022-09-27T15:55:55.3044809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnisvj9k_ 2022-09-27T15:55:55.3046292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnisvj9k_/_remote_module_non_scriptable.py 2022-09-27T15:55:55.3568698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ai5dw1e 2022-09-27T15:55:55.3569363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ai5dw1e/_remote_module_non_scriptable.py 2022-09-27T15:55:55.5236884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:55:55.5684386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:55.5795479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:55.6249454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:55:56.3450011Z ok (2.660s) 2022-09-27T15:55:56.3450206Z 2022-09-27T15:55:56.3450700Z ---------------------------------------------------------------------- 2022-09-27T15:55:56.3451107Z Ran 1 test in 2.661s 2022-09-27T15:55:56.3451209Z 2022-09-27T15:55:56.3451268Z OK 2022-09-27T15:55:56.3451356Z 2022-09-27T15:55:56.3451449Z Generating XML reports... 2022-09-27T15:55:56.3486458Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155553.xml 2022-09-27T15:55:57.7141848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:57.7142233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:57.7150100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:57.7150516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:57.8694186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwz6zqcv0 2022-09-27T15:55:57.8695769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwz6zqcv0/_remote_module_non_scriptable.py 2022-09-27T15:55:58.1278013Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:55:58.1288534Z 2022-09-27T15:55:58.1288662Z Running tests... 2022-09-27T15:55:58.1289230Z ---------------------------------------------------------------------- 2022-09-27T15:55:58.3691922Z test_wait_all (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12860 2022-09-27T15:55:58.3720077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12861 2022-09-27T15:55:58.3747975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12862 2022-09-27T15:55:58.3779060Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12863 2022-09-27T15:55:59.5343829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:59.5344496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:59.5345300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:59.5345942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:59.5418820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:59.5419355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:59.5732299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:59.5732903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:59.5735929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:59.5736492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:59.5806533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:59.5807197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:59.6036560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:55:59.6037186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:55:59.6412641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:55:59.6413262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:55:59.7018260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8b05dln 2022-09-27T15:55:59.7019235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8b05dln/_remote_module_non_scriptable.py 2022-09-27T15:55:59.7042829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpihu606lj 2022-09-27T15:55:59.7045084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpihu606lj/_remote_module_non_scriptable.py 2022-09-27T15:55:59.7110762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0gxc5g2 2022-09-27T15:55:59.7112076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0gxc5g2/_remote_module_non_scriptable.py 2022-09-27T15:55:59.7856020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ekx4om_ 2022-09-27T15:55:59.7856799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ekx4om_/_remote_module_non_scriptable.py 2022-09-27T15:55:59.9574393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:55:59.9593396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:55:59.9699556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:00.0433972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:00.6835732Z ok (2.554s) 2022-09-27T15:56:00.6835964Z 2022-09-27T15:56:00.6836380Z ---------------------------------------------------------------------- 2022-09-27T15:56:00.6836630Z Ran 1 test in 2.555s 2022-09-27T15:56:00.6836745Z 2022-09-27T15:56:00.6836809Z OK 2022-09-27T15:56:00.6836899Z 2022-09-27T15:56:00.6836996Z Generating XML reports... 2022-09-27T15:56:00.6872019Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155558.xml 2022-09-27T15:56:02.0841215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:02.0841872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:02.0850039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:02.0850506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:02.2394111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5r0ufc_k 2022-09-27T15:56:02.2395902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5r0ufc_k/_remote_module_non_scriptable.py 2022-09-27T15:56:02.5007102Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:56:02.5016866Z 2022-09-27T15:56:02.5016979Z Running tests... 2022-09-27T15:56:02.5017579Z ---------------------------------------------------------------------- 2022-09-27T15:56:02.7512332Z test_wait_all_exit_early_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13083 2022-09-27T15:56:02.7545884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13084 2022-09-27T15:56:02.7577941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13085 2022-09-27T15:56:02.7609269Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13086 2022-09-27T15:56:03.8439806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:03.8440430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:03.8523738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:03.8646116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:03.8646892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:03.8647364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:03.8753395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:03.8753943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:03.8838705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:03.8929189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:03.8930118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:03.8930755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:03.9054029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:03.9054705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:03.9148851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:03.9149506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:04.0113852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjh5v555 2022-09-27T15:56:04.0114814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjh5v555/_remote_module_non_scriptable.py 2022-09-27T15:56:04.0200195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3hzy4ti0 2022-09-27T15:56:04.0201466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3hzy4ti0/_remote_module_non_scriptable.py 2022-09-27T15:56:04.0322097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa45_krim 2022-09-27T15:56:04.0323090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa45_krim/_remote_module_non_scriptable.py 2022-09-27T15:56:04.0406181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplirw13l6 2022-09-27T15:56:04.0407544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplirw13l6/_remote_module_non_scriptable.py 2022-09-27T15:56:04.2658115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:04.2755792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:56:04.2880463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:56:04.2952522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:04.5512099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:56:04.5512962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:56:04.5618200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:56:04.5619043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:56:04.5620304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:04.5623036Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:04.5624200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:04.5625311Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:05.0666766Z ok (2.565s) 2022-09-27T15:56:05.0666956Z 2022-09-27T15:56:05.0667269Z ---------------------------------------------------------------------- 2022-09-27T15:56:05.0667572Z Ran 1 test in 2.565s 2022-09-27T15:56:05.0667689Z 2022-09-27T15:56:05.0667747Z OK 2022-09-27T15:56:05.0667838Z 2022-09-27T15:56:05.0667933Z Generating XML reports... 2022-09-27T15:56:05.0702999Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155602.xml 2022-09-27T15:56:06.4437375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:06.4437769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:06.4445581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:06.4446017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:06.6005746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9g7b1h_q 2022-09-27T15:56:06.6006276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9g7b1h_q/_remote_module_non_scriptable.py 2022-09-27T15:56:06.8600674Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:56:06.8609939Z 2022-09-27T15:56:06.8610231Z Running tests... 2022-09-27T15:56:06.8610819Z ---------------------------------------------------------------------- 2022-09-27T15:56:07.1050540Z test_wait_all_exit_early_python (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13318 2022-09-27T15:56:07.1078671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13319 2022-09-27T15:56:07.1106476Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13320 2022-09-27T15:56:07.1136123Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13321 2022-09-27T15:56:08.1956108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:08.1956756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:08.2132770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:08.2133453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:08.2216260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:08.2217104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:08.2261577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:08.2262334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:08.2347265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:08.2347826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:08.2531900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:08.2532519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:08.2611213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:08.2611919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:08.2649758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:08.2650342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:08.3626350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnmx5_41 2022-09-27T15:56:08.3627154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnmx5_41/_remote_module_non_scriptable.py 2022-09-27T15:56:08.3795941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpla5z2ne6 2022-09-27T15:56:08.3796909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpla5z2ne6/_remote_module_non_scriptable.py 2022-09-27T15:56:08.3884058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22mgdjst 2022-09-27T15:56:08.3885222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22mgdjst/_remote_module_non_scriptable.py 2022-09-27T15:56:08.3917108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumyh6c7o 2022-09-27T15:56:08.3918805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumyh6c7o/_remote_module_non_scriptable.py 2022-09-27T15:56:08.6201560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:56:08.6306495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:56:08.6411795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:08.6457087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:08.8868066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:56:08.8969311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:56:08.8970152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:56:08.8971378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:08.8972540Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:08.8973400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:56:08.8974466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:08.8975582Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:08.9098201Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:08.9098650Z ValueError('Expected error') 2022-09-27T15:56:08.9098945Z Traceback (most recent call last): 2022-09-27T15:56:08.9101182Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:08.9101613Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:08.9102158Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:08.9102569Z raise ValueError(expected_err) 2022-09-27T15:56:08.9102903Z ValueError: Expected error 2022-09-27T15:56:08.9103108Z 2022-09-27T15:56:08.9103260Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:08.9103650Z ValueError('Expected error') 2022-09-27T15:56:08.9103975Z Traceback (most recent call last): 2022-09-27T15:56:08.9104600Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:08.9105093Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:08.9105771Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:08.9106171Z raise ValueError(expected_err) 2022-09-27T15:56:08.9106484Z ValueError: Expected error 2022-09-27T15:56:08.9106675Z 2022-09-27T15:56:08.9106792Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:08.9107188Z ValueError('Expected error') 2022-09-27T15:56:08.9107536Z Traceback (most recent call last): 2022-09-27T15:56:08.9108159Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:08.9108677Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:08.9109385Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:08.9109776Z raise ValueError(expected_err) 2022-09-27T15:56:08.9109973Z ValueError: Expected error 2022-09-27T15:56:08.9110098Z 2022-09-27T15:56:08.9110211Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:08.9110455Z ValueError('Expected error') 2022-09-27T15:56:08.9110652Z Traceback (most recent call last): 2022-09-27T15:56:08.9111291Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:08.9111635Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:08.9112077Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:08.9112363Z raise ValueError(expected_err) 2022-09-27T15:56:08.9112565Z ValueError: Expected error 2022-09-27T15:56:08.9112683Z 2022-09-27T15:56:08.9176520Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:08.9177041Z ValueError('Expected error') 2022-09-27T15:56:08.9177379Z Traceback (most recent call last): 2022-09-27T15:56:08.9178043Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:08.9178627Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:08.9179350Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:08.9179825Z raise ValueError(expected_err) 2022-09-27T15:56:08.9180132Z ValueError: Expected error 2022-09-27T15:56:08.9180309Z 2022-09-27T15:56:08.9180451Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:08.9180728Z ValueError('Expected error') 2022-09-27T15:56:08.9180938Z Traceback (most recent call last): 2022-09-27T15:56:08.9181316Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:08.9181649Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:08.9182090Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:08.9182392Z raise ValueError(expected_err) 2022-09-27T15:56:08.9182584Z ValueError: Expected error 2022-09-27T15:56:08.9182879Z 2022-09-27T15:56:08.9218318Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:08.9219906Z ValueError('Expected error') 2022-09-27T15:56:08.9220665Z Traceback (most recent call last): 2022-09-27T15:56:08.9221566Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:08.9222269Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:08.9223232Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:08.9223823Z raise ValueError(expected_err) 2022-09-27T15:56:08.9224270Z ValueError: Expected error 2022-09-27T15:56:08.9224434Z 2022-09-27T15:56:08.9226291Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:08.9226953Z ValueError('Expected error') 2022-09-27T15:56:08.9227510Z Traceback (most recent call last): 2022-09-27T15:56:08.9228383Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:08.9229099Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:08.9229803Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:08.9230317Z raise ValueError(expected_err) 2022-09-27T15:56:08.9230633Z ValueError: Expected error 2022-09-27T15:56:08.9230826Z 2022-09-27T15:56:09.3192711Z ok (2.458s) 2022-09-27T15:56:09.3192941Z 2022-09-27T15:56:09.3193456Z ---------------------------------------------------------------------- 2022-09-27T15:56:09.3193834Z Ran 1 test in 2.458s 2022-09-27T15:56:09.3193951Z 2022-09-27T15:56:09.3194010Z OK 2022-09-27T15:56:09.3194102Z 2022-09-27T15:56:09.3194183Z Generating XML reports... 2022-09-27T15:56:09.3231008Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155606.xml 2022-09-27T15:56:10.6919980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:10.6920360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:10.6930055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:10.6930404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:10.8533180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4br7uam 2022-09-27T15:56:10.8534728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4br7uam/_remote_module_non_scriptable.py 2022-09-27T15:56:11.1219780Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:56:11.1229304Z 2022-09-27T15:56:11.1229441Z Running tests... 2022-09-27T15:56:11.1230017Z ---------------------------------------------------------------------- 2022-09-27T15:56:11.3740459Z test_wait_all_exit_early_script_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13553 2022-09-27T15:56:11.3768144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13554 2022-09-27T15:56:11.3795271Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13555 2022-09-27T15:56:11.3825921Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13556 2022-09-27T15:56:12.5036056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:12.5036698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:12.5117639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:12.5118279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:12.5406712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:12.5407415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:12.5425365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:12.5425950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:12.5483931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:12.5484546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:12.5485272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:12.5485819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:12.5801270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:12.5801914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:12.5869287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:12.5869921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:12.6678373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl5fa2iq7 2022-09-27T15:56:12.6679065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl5fa2iq7/_remote_module_non_scriptable.py 2022-09-27T15:56:12.6748615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpay2dtxs1 2022-09-27T15:56:12.6752013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpay2dtxs1/_remote_module_non_scriptable.py 2022-09-27T15:56:12.7060042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6l2dtrjh 2022-09-27T15:56:12.7060799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6l2dtrjh/_remote_module_non_scriptable.py 2022-09-27T15:56:12.7137098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph41gmnzl 2022-09-27T15:56:12.7138300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph41gmnzl/_remote_module_non_scriptable.py 2022-09-27T15:56:12.9219479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:56:12.9295521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:12.9606659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:56:12.9702917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:13.2386231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:56:13.2487260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:56:13.2491432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:56:13.2492264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:56:13.2493509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:13.2494642Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:13.2495844Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:13.2496978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:13.7884058Z ok (2.665s) 2022-09-27T15:56:13.7884482Z 2022-09-27T15:56:13.7885371Z ---------------------------------------------------------------------- 2022-09-27T15:56:13.7885626Z Ran 1 test in 2.665s 2022-09-27T15:56:13.7885814Z 2022-09-27T15:56:13.7885875Z OK 2022-09-27T15:56:13.7885968Z 2022-09-27T15:56:13.7886063Z Generating XML reports... 2022-09-27T15:56:13.7922200Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155611.xml 2022-09-27T15:56:15.1266838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:15.1267264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:15.1275128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:15.1275491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:15.2834185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpysojdc77 2022-09-27T15:56:15.2834859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpysojdc77/_remote_module_non_scriptable.py 2022-09-27T15:56:15.5452598Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:56:15.5462473Z 2022-09-27T15:56:15.5462575Z Running tests... 2022-09-27T15:56:15.5463551Z ---------------------------------------------------------------------- 2022-09-27T15:56:15.7888174Z test_wait_all_multiple_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13788 2022-09-27T15:56:15.7915623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13789 2022-09-27T15:56:15.7943222Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13790 2022-09-27T15:56:15.7973986Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13791 2022-09-27T15:56:16.8901871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:16.8902479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:16.8980659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:16.8981327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:16.9265719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:16.9266346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:16.9350992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:16.9351731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:16.9936450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:16.9937124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:17.0185440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:17.0186053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:17.0285803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:17.0286464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:17.0525845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:17.0526425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:17.0531960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpslv1xrac 2022-09-27T15:56:17.0533731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpslv1xrac/_remote_module_non_scriptable.py 2022-09-27T15:56:17.0611992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ayuz4a7 2022-09-27T15:56:17.0613993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ayuz4a7/_remote_module_non_scriptable.py 2022-09-27T15:56:17.1546518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7yggjjx 2022-09-27T15:56:17.1547342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7yggjjx/_remote_module_non_scriptable.py 2022-09-27T15:56:17.1785241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumozoid8 2022-09-27T15:56:17.1786022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumozoid8/_remote_module_non_scriptable.py 2022-09-27T15:56:17.3087331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:56:17.3163912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:56:17.4097138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:17.4546463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:18.1032251Z ok (2.557s) 2022-09-27T15:56:18.1032474Z 2022-09-27T15:56:18.1032975Z ---------------------------------------------------------------------- 2022-09-27T15:56:18.1033370Z Ran 1 test in 2.557s 2022-09-27T15:56:18.1033493Z 2022-09-27T15:56:18.1033552Z OK 2022-09-27T15:56:18.1033647Z 2022-09-27T15:56:18.1033742Z Generating XML reports... 2022-09-27T15:56:18.1067922Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155615.xml 2022-09-27T15:56:19.4594007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:19.4594800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:19.4602916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:19.4603572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:19.6160558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvt67pe3 2022-09-27T15:56:19.6161523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvt67pe3/_remote_module_non_scriptable.py 2022-09-27T15:56:19.8750284Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:56:19.8760146Z 2022-09-27T15:56:19.8760629Z Running tests... 2022-09-27T15:56:19.8761215Z ---------------------------------------------------------------------- 2022-09-27T15:56:20.1218519Z test_wait_all_raise_in_body (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14011 2022-09-27T15:56:20.1246367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14012 2022-09-27T15:56:20.1274161Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14013 2022-09-27T15:56:20.1303883Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14014 2022-09-27T15:56:21.2480754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:21.2481382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:21.2785788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:21.2786453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:21.2788560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:21.2789472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:21.2837333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:21.2838330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:21.3171927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:21.3172547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:21.3174504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:21.3175149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:21.3217238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:21.3571306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:21.3572257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:21.3572889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:21.4108865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpapkajdg_ 2022-09-27T15:56:21.4110240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpapkajdg_/_remote_module_non_scriptable.py 2022-09-27T15:56:21.4452376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp746137if 2022-09-27T15:56:21.4453220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp746137if/_remote_module_non_scriptable.py 2022-09-27T15:56:21.4627278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7l9u46m3 2022-09-27T15:56:21.4628856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7l9u46m3/_remote_module_non_scriptable.py 2022-09-27T15:56:21.4834467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7iqtpn0 2022-09-27T15:56:21.4835974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7iqtpn0/_remote_module_non_scriptable.py 2022-09-27T15:56:21.6700346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:56:21.7050390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:21.7200708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:56:21.7425475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:22.3360264Z ok (2.460s) 2022-09-27T15:56:22.3360554Z 2022-09-27T15:56:22.3361032Z ---------------------------------------------------------------------- 2022-09-27T15:56:22.3361269Z Ran 1 test in 2.460s 2022-09-27T15:56:22.3361383Z 2022-09-27T15:56:22.3361443Z OK 2022-09-27T15:56:22.3361535Z 2022-09-27T15:56:22.3361639Z Generating XML reports... 2022-09-27T15:56:22.3396069Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155619.xml 2022-09-27T15:56:23.7112462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:23.7113083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:23.7122141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:23.7122790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:23.8726883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpclyfq97_ 2022-09-27T15:56:23.8727548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpclyfq97_/_remote_module_non_scriptable.py 2022-09-27T15:56:24.1389705Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:56:24.1399935Z 2022-09-27T15:56:24.1400421Z Running tests... 2022-09-27T15:56:24.1401011Z ---------------------------------------------------------------------- 2022-09-27T15:56:24.3820806Z test_wait_all_raise_in_user_func (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14234 2022-09-27T15:56:24.3848185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14235 2022-09-27T15:56:24.3878658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14236 2022-09-27T15:56:24.3907607Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14237 2022-09-27T15:56:25.5151782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:25.5152397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:25.5283508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:25.5284179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:25.5476240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:25.5476968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:25.5514792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:25.5515442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:25.5656882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:25.5657546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:25.5886281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:25.5886930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:25.5916780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:25.5917384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:25.6256469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:25.6257143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:25.6803743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3u7bxb_ 2022-09-27T15:56:25.6804519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3u7bxb_/_remote_module_non_scriptable.py 2022-09-27T15:56:25.6923857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmn5jzv3w 2022-09-27T15:56:25.6925045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmn5jzv3w/_remote_module_non_scriptable.py 2022-09-27T15:56:25.7146835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbkt398b 2022-09-27T15:56:25.7147647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbkt398b/_remote_module_non_scriptable.py 2022-09-27T15:56:25.7518151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1uxsutb 2022-09-27T15:56:25.7519496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1uxsutb/_remote_module_non_scriptable.py 2022-09-27T15:56:25.9353063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:56:25.9471327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:25.9717133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:26.0089879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:56:26.2624049Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:26.2624922Z ValueError('Expected error') 2022-09-27T15:56:26.2625232Z Traceback (most recent call last): 2022-09-27T15:56:26.2625880Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:26.2626453Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:26.2627166Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:26.2627636Z raise ValueError(expected_err) 2022-09-27T15:56:26.2627956Z ValueError: Expected error 2022-09-27T15:56:26.2628134Z 2022-09-27T15:56:26.2807064Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:26.2807669Z ValueError('Expected error') 2022-09-27T15:56:26.2808045Z Traceback (most recent call last): 2022-09-27T15:56:26.2808814Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:26.2809504Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:26.2810305Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:26.2810868Z raise ValueError(expected_err) 2022-09-27T15:56:26.2811229Z ValueError: Expected error 2022-09-27T15:56:26.2811453Z 2022-09-27T15:56:26.2825556Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:26.2826269Z ValueError('Expected error') 2022-09-27T15:56:26.2826668Z Traceback (most recent call last): 2022-09-27T15:56:26.2827465Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:26.2828163Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:26.2830491Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:26.2831585Z raise ValueError(expected_err) 2022-09-27T15:56:26.2832348Z ValueError: Expected error 2022-09-27T15:56:26.2832650Z 2022-09-27T15:56:26.2892404Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:26.2893859Z ValueError('Expected error') 2022-09-27T15:56:26.2894261Z Traceback (most recent call last): 2022-09-27T15:56:26.2894952Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:26.2895522Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:26.2896181Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:26.2896488Z raise ValueError(expected_err) 2022-09-27T15:56:26.2896685Z ValueError: Expected error 2022-09-27T15:56:26.2896809Z 2022-09-27T15:56:26.6964141Z ok (2.556s) 2022-09-27T15:56:26.6964363Z 2022-09-27T15:56:26.6964688Z ---------------------------------------------------------------------- 2022-09-27T15:56:26.6964940Z Ran 1 test in 2.556s 2022-09-27T15:56:26.6965066Z 2022-09-27T15:56:26.6965128Z OK 2022-09-27T15:56:26.6965218Z 2022-09-27T15:56:26.6965311Z Generating XML reports... 2022-09-27T15:56:26.7000103Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155624.xml 2022-09-27T15:56:28.0690056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:28.0690418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:28.0698167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:28.0698526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:28.2247791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2c3_k1v 2022-09-27T15:56:28.2249515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2c3_k1v/_remote_module_non_scriptable.py 2022-09-27T15:56:28.4847005Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:56:28.4855903Z 2022-09-27T15:56:28.4856031Z Running tests... 2022-09-27T15:56:28.4856595Z ---------------------------------------------------------------------- 2022-09-27T15:56:28.7295200Z test_wait_all_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14457 2022-09-27T15:56:28.7321982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14458 2022-09-27T15:56:28.7350394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14459 2022-09-27T15:56:28.7379168Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14460 2022-09-27T15:56:29.9053420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:29.9054087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:29.9393479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:29.9394180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:29.9395220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:29.9395896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:29.9423433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:29.9424160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:29.9448886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:29.9449472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:29.9781310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:29.9781973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:29.9782980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:29.9783528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:29.9836147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:29.9836837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:30.0688331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphydju56q 2022-09-27T15:56:30.0689166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphydju56q/_remote_module_non_scriptable.py 2022-09-27T15:56:30.1048752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb035wuz3 2022-09-27T15:56:30.1049536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb035wuz3/_remote_module_non_scriptable.py 2022-09-27T15:56:30.1050635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvlzaaj4 2022-09-27T15:56:30.1053259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvlzaaj4/_remote_module_non_scriptable.py 2022-09-27T15:56:30.1089051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8m_6idp 2022-09-27T15:56:30.1090487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8m_6idp/_remote_module_non_scriptable.py 2022-09-27T15:56:30.3232031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:30.3603675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:30.3611543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:56:30.3627310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:56:30.7288884Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:30.7300969Z [W tensorpipe_agent.cpp:940] RPC agent for worker3 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:31.5871739Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #3 to worker0, as the agent is shutting down 2022-09-27T15:56:31.6072103Z [W tensorpipe_agent.cpp:625] RPC agent for worker0 won't send response to request #1 to worker3, as the agent is shutting down 2022-09-27T15:56:31.6197055Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #1 to worker2, as the agent is shutting down 2022-09-27T15:56:31.6235568Z [W tensorpipe_agent.cpp:625] RPC agent for worker2 won't send response to request #1 to worker1, as the agent is shutting down 2022-09-27T15:56:31.9451833Z ok (3.459s) 2022-09-27T15:56:31.9452053Z 2022-09-27T15:56:31.9452556Z ---------------------------------------------------------------------- 2022-09-27T15:56:31.9452915Z Ran 1 test in 3.459s 2022-09-27T15:56:31.9453019Z 2022-09-27T15:56:31.9453085Z OK 2022-09-27T15:56:31.9453179Z 2022-09-27T15:56:31.9453276Z Generating XML reports... 2022-09-27T15:56:31.9490515Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155628.xml 2022-09-27T15:56:33.3129265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:33.3129856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:33.3138152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:33.3138778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:33.4712309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_w_yj4h 2022-09-27T15:56:33.4712973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_w_yj4h/_remote_module_non_scriptable.py 2022-09-27T15:56:33.7312018Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:56:33.7322053Z 2022-09-27T15:56:33.7322498Z Running tests... 2022-09-27T15:56:33.7323144Z ---------------------------------------------------------------------- 2022-09-27T15:56:33.9738303Z test_wait_all_with_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14680 2022-09-27T15:56:33.9766438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14681 2022-09-27T15:56:33.9794021Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14682 2022-09-27T15:56:33.9823988Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14683 2022-09-27T15:56:35.0583097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:35.0583733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:35.0794971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:35.0795527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:35.0961557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:35.0962495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:35.1011734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:35.1012595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:35.1182022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:35.1182652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:35.1183335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:35.1183855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:35.1394782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:35.1395396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:35.1548473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:35.1549108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:35.2231857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hyyq6dm 2022-09-27T15:56:35.2232723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hyyq6dm/_remote_module_non_scriptable.py 2022-09-27T15:56:35.2462909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7trbxwic 2022-09-27T15:56:35.2464225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7trbxwic/_remote_module_non_scriptable.py 2022-09-27T15:56:35.2646818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakdrdsmt 2022-09-27T15:56:35.2647649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakdrdsmt/_remote_module_non_scriptable.py 2022-09-27T15:56:35.2804982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8f6t05m 2022-09-27T15:56:35.2805730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8f6t05m/_remote_module_non_scriptable.py 2022-09-27T15:56:35.4794370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:35.5032466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:56:35.5182817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:56:35.5338474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:35.7482547Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:35.7483281Z ValueError('Expected error') 2022-09-27T15:56:35.7483722Z Traceback (most recent call last): 2022-09-27T15:56:35.7484620Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7485333Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7486306Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7486936Z raise ValueError(expected_err) 2022-09-27T15:56:35.7487378Z ValueError: Expected error 2022-09-27T15:56:35.7495006Z 2022-09-27T15:56:35.7495569Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:35.7496224Z ValueError('Expected error') 2022-09-27T15:56:35.7496707Z Traceback (most recent call last): 2022-09-27T15:56:35.7497484Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7498214Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7499165Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7500179Z raise ValueError(expected_err) 2022-09-27T15:56:35.7500634Z ValueError: Expected error 2022-09-27T15:56:35.7500904Z 2022-09-27T15:56:35.7501236Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:35.7501740Z ValueError('Expected error') 2022-09-27T15:56:35.7502222Z Traceback (most recent call last): 2022-09-27T15:56:35.7503071Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7503767Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7504690Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7505356Z raise ValueError(expected_err) 2022-09-27T15:56:35.7505747Z ValueError: Expected error 2022-09-27T15:56:35.7505995Z 2022-09-27T15:56:35.7506194Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:35.7513805Z ValueError('Expected error') 2022-09-27T15:56:35.7514441Z Traceback (most recent call last): 2022-09-27T15:56:35.7515764Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7516685Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7517632Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7518281Z raise ValueError(expected_err) 2022-09-27T15:56:35.7518725Z ValueError: Expected error 2022-09-27T15:56:35.7518983Z 2022-09-27T15:56:35.7519146Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:35.7519660Z ValueError('Expected error') 2022-09-27T15:56:35.7520107Z Traceback (most recent call last): 2022-09-27T15:56:35.7520931Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7521614Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7522570Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7523215Z raise ValueError(expected_err) 2022-09-27T15:56:35.7523631Z ValueError: Expected error 2022-09-27T15:56:35.7523892Z 2022-09-27T15:56:35.7524082Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:35.7524587Z ValueError('Expected error') 2022-09-27T15:56:35.7525013Z Traceback (most recent call last): 2022-09-27T15:56:35.7525834Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7526534Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7527478Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7528110Z raise ValueError(expected_err) 2022-09-27T15:56:35.7528549Z ValueError: Expected error 2022-09-27T15:56:35.7528801Z 2022-09-27T15:56:35.7528995Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:35.7529479Z ValueError('Expected error') 2022-09-27T15:56:35.7529922Z Traceback (most recent call last): 2022-09-27T15:56:35.7530751Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7531428Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7532377Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7533022Z raise ValueError(expected_err) 2022-09-27T15:56:35.7533466Z ValueError: Expected error 2022-09-27T15:56:35.7533703Z 2022-09-27T15:56:35.7533889Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:35.7534390Z ValueError('Expected error') 2022-09-27T15:56:35.7534838Z Traceback (most recent call last): 2022-09-27T15:56:35.7535632Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7536576Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7537539Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7538300Z raise ValueError(expected_err) 2022-09-27T15:56:35.7538712Z ValueError: Expected error 2022-09-27T15:56:35.7538971Z 2022-09-27T15:56:35.7539167Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:35.7539673Z ValueError('Expected error') 2022-09-27T15:56:35.7540096Z Traceback (most recent call last): 2022-09-27T15:56:35.7540915Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7541619Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7542545Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7543196Z raise ValueError(expected_err) 2022-09-27T15:56:35.7543634Z ValueError: Expected error 2022-09-27T15:56:35.7543894Z 2022-09-27T15:56:35.7544079Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:35.7544570Z ValueError('Expected error') 2022-09-27T15:56:35.7545014Z Traceback (most recent call last): 2022-09-27T15:56:35.7545834Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7546506Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7547599Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7548247Z raise ValueError(expected_err) 2022-09-27T15:56:35.7548668Z ValueError: Expected error 2022-09-27T15:56:35.7548921Z 2022-09-27T15:56:35.7627956Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:35.7628519Z ValueError('Expected error') 2022-09-27T15:56:35.7628908Z Traceback (most recent call last): 2022-09-27T15:56:35.7629643Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7630149Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7630611Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7630907Z raise ValueError(expected_err) 2022-09-27T15:56:35.7631263Z ValueError: Expected error 2022-09-27T15:56:35.7636735Z 2022-09-27T15:56:35.7636974Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:35.7637488Z ValueError('Expected error') 2022-09-27T15:56:35.7637860Z Traceback (most recent call last): 2022-09-27T15:56:35.7638596Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7639198Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7639767Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7640117Z raise ValueError(expected_err) 2022-09-27T15:56:35.7640333Z ValueError: Expected error 2022-09-27T15:56:35.7640800Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:35.7641213Z ValueError('Expected error') 2022-09-27T15:56:35.7642439Z Traceback (most recent call last): 2022-09-27T15:56:35.7646873Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7647455Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7648164Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7648625Z raise ValueError(expected_err) 2022-09-27T15:56:35.7648944Z ValueError: Expected error 2022-09-27T15:56:35.7649142Z 2022-09-27T15:56:35.7649289Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:35.7649640Z ValueError('Expected error') 2022-09-27T15:56:35.7650154Z Traceback (most recent call last): 2022-09-27T15:56:35.7650766Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7651313Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7656343Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7656864Z raise ValueError(expected_err) 2022-09-27T15:56:35.7657202Z ValueError: Expected error 2022-09-27T15:56:35.7661086Z 2022-09-27T15:56:35.7661253Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:35.7661712Z ValueError('Expected error') 2022-09-27T15:56:35.7662049Z Traceback (most recent call last): 2022-09-27T15:56:35.7662692Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7663238Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7663992Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7664502Z raise ValueError(expected_err) 2022-09-27T15:56:35.7664852Z ValueError: Expected error 2022-09-27T15:56:35.7665081Z 2022-09-27T15:56:35.7665255Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:35.7665705Z ValueError('Expected error') 2022-09-27T15:56:35.7666080Z Traceback (most recent call last): 2022-09-27T15:56:35.7666807Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7667418Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7668136Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7668661Z raise ValueError(expected_err) 2022-09-27T15:56:35.7669051Z ValueError: Expected error 2022-09-27T15:56:35.7669272Z 2022-09-27T15:56:35.7670766Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:35.7673152Z ValueError('Expected error') 2022-09-27T15:56:35.7673507Z Traceback (most recent call last): 2022-09-27T15:56:35.7674453Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7675118Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7675827Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7676340Z raise ValueError(expected_err) 2022-09-27T15:56:35.7676667Z ValueError: Expected error 2022-09-27T15:56:35.7676976Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:35.7677344Z ValueError('Expected error') 2022-09-27T15:56:35.7677665Z Traceback (most recent call last): 2022-09-27T15:56:35.7678252Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7678769Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7679487Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7679956Z raise ValueError(expected_err) 2022-09-27T15:56:35.7680268Z ValueError: Expected error 2022-09-27T15:56:35.7680458Z 2022-09-27T15:56:35.7680464Z 2022-09-27T15:56:35.7680593Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:35.7680998Z ValueError('Expected error') 2022-09-27T15:56:35.7681328Z Traceback (most recent call last): 2022-09-27T15:56:35.7681983Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7682524Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7683647Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7684388Z raise ValueError(expected_err) 2022-09-27T15:56:35.7684727Z ValueError: Expected error 2022-09-27T15:56:35.7685008Z 2022-09-27T15:56:35.7685137Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:35.7685523Z ValueError('Expected error') 2022-09-27T15:56:35.7685863Z Traceback (most recent call last): 2022-09-27T15:56:35.7686510Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7687039Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7687734Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7688227Z raise ValueError(expected_err) 2022-09-27T15:56:35.7688550Z ValueError: Expected error 2022-09-27T15:56:35.7688856Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:35.7689249Z ValueError('Expected error') 2022-09-27T15:56:35.7689581Z Traceback (most recent call last): 2022-09-27T15:56:35.7690187Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7690729Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7691449Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7691937Z raise ValueError(expected_err) 2022-09-27T15:56:35.7692234Z ValueError: Expected error 2022-09-27T15:56:35.7692425Z 2022-09-27T15:56:35.7692562Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:35.7692947Z ValueError('Expected error') 2022-09-27T15:56:35.7693261Z Traceback (most recent call last): 2022-09-27T15:56:35.7693866Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7694376Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7695067Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7695551Z raise ValueError(expected_err) 2022-09-27T15:56:35.7695916Z ValueError: Expected error 2022-09-27T15:56:35.7696450Z 2022-09-27T15:56:35.7696983Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:35.7697420Z ValueError('Expected error') 2022-09-27T15:56:35.7697758Z Traceback (most recent call last): 2022-09-27T15:56:35.7698509Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7699132Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7699601Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7699895Z raise ValueError(expected_err) 2022-09-27T15:56:35.7700105Z ValueError: Expected error 2022-09-27T15:56:35.7700390Z 2022-09-27T15:56:35.7700483Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:35.7700719Z ValueError('Expected error') 2022-09-27T15:56:35.7700938Z Traceback (most recent call last): 2022-09-27T15:56:35.7701328Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7701668Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7702098Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7702403Z raise ValueError(expected_err) 2022-09-27T15:56:35.7702612Z ValueError: Expected error 2022-09-27T15:56:35.7702734Z 2022-09-27T15:56:35.7702811Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:35.7703053Z ValueError('Expected error') 2022-09-27T15:56:35.7703262Z Traceback (most recent call last): 2022-09-27T15:56:35.7703632Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7704180Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7704665Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7705016Z raise ValueError(expected_err) 2022-09-27T15:56:35.7705224Z ValueError: Expected error 2022-09-27T15:56:35.7705399Z 2022-09-27T15:56:35.7705529Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:35.7705860Z ValueError('Expected error') 2022-09-27T15:56:35.7706167Z Traceback (most recent call last): 2022-09-27T15:56:35.7706679Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7707177Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7707873Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7708383Z raise ValueError(expected_err) 2022-09-27T15:56:35.7708754Z ValueError: Expected error 2022-09-27T15:56:35.7708968Z 2022-09-27T15:56:35.7709133Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:35.7709561Z ValueError('Expected error') 2022-09-27T15:56:35.7709948Z Traceback (most recent call last): 2022-09-27T15:56:35.7710646Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7711310Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7711808Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7712114Z raise ValueError(expected_err) 2022-09-27T15:56:35.7712323Z ValueError: Expected error 2022-09-27T15:56:35.7712433Z 2022-09-27T15:56:35.7712452Z 2022-09-27T15:56:35.7712544Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:35.7712786Z ValueError('Expected error') 2022-09-27T15:56:35.7712997Z Traceback (most recent call last): 2022-09-27T15:56:35.7713367Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7713759Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7714203Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7714652Z raise ValueError(expected_err) 2022-09-27T15:56:35.7714862Z ValueError: Expected error 2022-09-27T15:56:35.7715004Z 2022-09-27T15:56:35.7715010Z 2022-09-27T15:56:35.7719730Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:35.7721836Z ValueError('Expected error') 2022-09-27T15:56:35.7722313Z Traceback (most recent call last): 2022-09-27T15:56:35.7723173Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7723904Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7724866Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7725497Z raise ValueError(expected_err) 2022-09-27T15:56:35.7725934Z ValueError: Expected error 2022-09-27T15:56:35.7726191Z 2022-09-27T15:56:35.7737998Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:35.7755512Z ValueError('Expected error') 2022-09-27T15:56:35.7755935Z Traceback (most recent call last): 2022-09-27T15:56:35.7756622Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7757245Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7758074Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7758394Z raise ValueError(expected_err) 2022-09-27T15:56:35.7758604Z ValueError: Expected error 2022-09-27T15:56:35.7758713Z 2022-09-27T15:56:35.7784366Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:35.7784985Z ValueError('Expected error') 2022-09-27T15:56:35.7785440Z Traceback (most recent call last): 2022-09-27T15:56:35.7786358Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7787070Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7788032Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7788682Z raise ValueError(expected_err) 2022-09-27T15:56:35.7789108Z ValueError: Expected error 2022-09-27T15:56:35.7791664Z 2022-09-27T15:56:35.7792042Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:35.7792598Z ValueError('Expected error') 2022-09-27T15:56:35.7793037Z Traceback (most recent call last): 2022-09-27T15:56:35.7793858Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7794735Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7795697Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7796329Z raise ValueError(expected_err) 2022-09-27T15:56:35.7796774Z ValueError: Expected error 2022-09-27T15:56:35.7797205Z 2022-09-27T15:56:35.7797400Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:35.7798138Z ValueError('Expected error') 2022-09-27T15:56:35.7799474Z Traceback (most recent call last): 2022-09-27T15:56:35.7800310Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7800999Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7801960Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7802610Z raise ValueError(expected_err) 2022-09-27T15:56:35.7803030Z ValueError: Expected error 2022-09-27T15:56:35.7803284Z 2022-09-27T15:56:35.7804004Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:35.7804368Z ValueError('Expected error') 2022-09-27T15:56:35.7804700Z Traceback (most recent call last): 2022-09-27T15:56:35.7806718Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7807424Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7808363Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7809007Z raise ValueError(expected_err) 2022-09-27T15:56:35.7809457Z ValueError: Expected error 2022-09-27T15:56:35.7809727Z 2022-09-27T15:56:35.7809901Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:35.7811064Z ValueError('Expected error') 2022-09-27T15:56:35.7811516Z Traceback (most recent call last): 2022-09-27T15:56:35.7812357Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7813049Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7814028Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7814673Z raise ValueError(expected_err) 2022-09-27T15:56:35.7815108Z ValueError: Expected error 2022-09-27T15:56:35.7815363Z 2022-09-27T15:56:35.7815533Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:35.7816042Z ValueError('Expected error') 2022-09-27T15:56:35.7816487Z Traceback (most recent call last): 2022-09-27T15:56:35.7817281Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7817974Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7819075Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7819725Z raise ValueError(expected_err) 2022-09-27T15:56:35.7820238Z ValueError: Expected error 2022-09-27T15:56:35.7820493Z 2022-09-27T15:56:35.7820683Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:35.7821187Z ValueError('Expected error') 2022-09-27T15:56:35.7821615Z Traceback (most recent call last): 2022-09-27T15:56:35.7822430Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7823130Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7824057Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7824706Z raise ValueError(expected_err) 2022-09-27T15:56:35.7825140Z ValueError: Expected error 2022-09-27T15:56:35.7825353Z 2022-09-27T15:56:35.7825485Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:35.7825853Z ValueError('Expected error') 2022-09-27T15:56:35.7826297Z Traceback (most recent call last): 2022-09-27T15:56:35.7827118Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7827795Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7828686Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7829316Z raise ValueError(expected_err) 2022-09-27T15:56:35.7829686Z ValueError: Expected error 2022-09-27T15:56:35.7829892Z 2022-09-27T15:56:35.7830077Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:35.7830533Z ValueError('Expected error') 2022-09-27T15:56:35.7830973Z Traceback (most recent call last): 2022-09-27T15:56:35.7831742Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7832382Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7833344Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7833980Z raise ValueError(expected_err) 2022-09-27T15:56:35.7834420Z ValueError: Expected error 2022-09-27T15:56:35.7834793Z 2022-09-27T15:56:35.7834988Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:35.7835506Z ValueError('Expected error') 2022-09-27T15:56:35.7835934Z Traceback (most recent call last): 2022-09-27T15:56:35.7836773Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:35.7837480Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:35.7838414Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:35.7839057Z raise ValueError(expected_err) 2022-09-27T15:56:35.7839503Z ValueError: Expected error 2022-09-27T15:56:35.7839760Z 2022-09-27T15:56:36.1880526Z ok (2.455s) 2022-09-27T15:56:36.1880776Z 2022-09-27T15:56:36.1881247Z ---------------------------------------------------------------------- 2022-09-27T15:56:36.1881516Z Ran 1 test in 2.456s 2022-09-27T15:56:36.1881634Z 2022-09-27T15:56:36.1881695Z OK 2022-09-27T15:56:36.1881789Z 2022-09-27T15:56:36.1881871Z Generating XML reports... 2022-09-27T15:56:36.1917769Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155633.xml 2022-09-27T15:56:37.5758166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:37.5758538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:37.5766794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:37.5767408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:37.7363295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp132ch02s 2022-09-27T15:56:37.7364501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp132ch02s/_remote_module_non_scriptable.py 2022-09-27T15:56:37.9996867Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:56:38.0006206Z 2022-09-27T15:56:38.0006397Z Running tests... 2022-09-27T15:56:38.0006740Z ---------------------------------------------------------------------- 2022-09-27T15:56:38.2432923Z test_wait_all_with_partial_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14903 2022-09-27T15:56:38.2461772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14904 2022-09-27T15:56:38.2489988Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14905 2022-09-27T15:56:38.2519711Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14906 2022-09-27T15:56:39.3417433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:39.3418111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:39.3656260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:39.3656858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:39.3778232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:39.3779005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:39.3779874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:39.3780539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:39.3971754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:39.3972453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:39.4051504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:39.4052057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:39.4173040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:39.4173678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:39.4354297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:39.4355128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:39.5065864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvuux4690 2022-09-27T15:56:39.5066688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvuux4690/_remote_module_non_scriptable.py 2022-09-27T15:56:39.5319565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23rqv58t 2022-09-27T15:56:39.5320649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23rqv58t/_remote_module_non_scriptable.py 2022-09-27T15:56:39.5444267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp630d2k4s 2022-09-27T15:56:39.5445659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp630d2k4s/_remote_module_non_scriptable.py 2022-09-27T15:56:39.5619169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl35xoypr 2022-09-27T15:56:39.5620644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl35xoypr/_remote_module_non_scriptable.py 2022-09-27T15:56:39.7633021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:56:39.7898603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:56:39.8007060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:39.8201580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:40.0264965Z On WorkerInfo(id=1, name=worker1): 2022-09-27T15:56:40.0265763Z ValueError('Expected error') 2022-09-27T15:56:40.0266231Z Traceback (most recent call last): 2022-09-27T15:56:40.0267091Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:40.0267813Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:40.0268787Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:40.0322141Z raise ValueError(expected_err) 2022-09-27T15:56:40.0322576Z ValueError: Expected error 2022-09-27T15:56:40.0322800Z 2022-09-27T15:56:40.0419018Z On WorkerInfo(id=2, name=worker2): 2022-09-27T15:56:40.0419789Z ValueError('Expected error') 2022-09-27T15:56:40.0420034Z Traceback (most recent call last): 2022-09-27T15:56:40.0422693Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:40.0423460Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:40.0424557Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:40.0425243Z raise ValueError(expected_err) 2022-09-27T15:56:40.0425699Z ValueError: Expected error 2022-09-27T15:56:40.0431403Z 2022-09-27T15:56:40.0471721Z On WorkerInfo(id=0, name=worker0): 2022-09-27T15:56:40.0472198Z ValueError('Expected error') 2022-09-27T15:56:40.0472443Z Traceback (most recent call last): 2022-09-27T15:56:40.0472845Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:40.0473190Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:40.0473639Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:40.0473937Z raise ValueError(expected_err) 2022-09-27T15:56:40.0474150Z ValueError: Expected error 2022-09-27T15:56:40.0474273Z 2022-09-27T15:56:40.0516577Z On WorkerInfo(id=3, name=worker3): 2022-09-27T15:56:40.0517030Z ValueError('Expected error') 2022-09-27T15:56:40.0517396Z Traceback (most recent call last): 2022-09-27T15:56:40.0518138Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-09-27T15:56:40.0518749Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-09-27T15:56:40.0519557Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-09-27T15:56:40.0520065Z raise ValueError(expected_err) 2022-09-27T15:56:40.0522457Z ValueError: Expected error 2022-09-27T15:56:40.0522989Z 2022-09-27T15:56:40.4576186Z ok (2.457s) 2022-09-27T15:56:40.4576453Z 2022-09-27T15:56:40.4576895Z ---------------------------------------------------------------------- 2022-09-27T15:56:40.4577162Z Ran 1 test in 2.457s 2022-09-27T15:56:40.4577281Z 2022-09-27T15:56:40.4577343Z OK 2022-09-27T15:56:40.4577437Z 2022-09-27T15:56:40.4577533Z Generating XML reports... 2022-09-27T15:56:40.4613038Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155637.xml 2022-09-27T15:56:41.8217670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:41.8218300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:41.8226382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:41.8226844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:41.9807963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7ph8dol 2022-09-27T15:56:41.9809646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7ph8dol/_remote_module_non_scriptable.py 2022-09-27T15:56:42.2438693Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:56:42.2447409Z 2022-09-27T15:56:42.2447523Z Running tests... 2022-09-27T15:56:42.2447989Z ---------------------------------------------------------------------- 2022-09-27T15:56:42.4899850Z test_wait_all_workers_dense (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15126 2022-09-27T15:56:42.4928564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15127 2022-09-27T15:56:42.4956964Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15128 2022-09-27T15:56:42.4986836Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15129 2022-09-27T15:56:43.5719625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:43.5720227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:43.5983939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:43.5984549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:43.6078112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:43.6078780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:43.6110611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:43.6111321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:43.6389800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:43.6390414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:43.6455054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:43.6455759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:43.6479050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:43.6479746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:43.6844909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:43.6845507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:43.7400869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1tn6sq53 2022-09-27T15:56:43.7401673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1tn6sq53/_remote_module_non_scriptable.py 2022-09-27T15:56:43.7668180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpth1bvo2q 2022-09-27T15:56:43.7669490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpth1bvo2q/_remote_module_non_scriptable.py 2022-09-27T15:56:43.7772832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1zo05iw 2022-09-27T15:56:43.7773838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1zo05iw/_remote_module_non_scriptable.py 2022-09-27T15:56:43.8135142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvq72kak 2022-09-27T15:56:43.8135911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvq72kak/_remote_module_non_scriptable.py 2022-09-27T15:56:44.0050172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:56:44.0304898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:44.0437973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:56:44.0761226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:44.1020036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:56:44.1121581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:56:44.1122183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:56:44.1122774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:56:44.1123709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:44.1124550Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:44.1125400Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:44.1126217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:44.5127454Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:56:44.5131527Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:56:44.5132539Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:56:44.5184833Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:56:44.5185927Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:56:44.5186989Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:56:44.9048192Z ok (2.660s) 2022-09-27T15:56:44.9048459Z 2022-09-27T15:56:44.9048910Z ---------------------------------------------------------------------- 2022-09-27T15:56:44.9049316Z Ran 1 test in 2.660s 2022-09-27T15:56:44.9049487Z 2022-09-27T15:56:44.9049576Z OK 2022-09-27T15:56:44.9049703Z 2022-09-27T15:56:44.9049845Z Generating XML reports... 2022-09-27T15:56:44.9085727Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155642.xml 2022-09-27T15:56:46.3305515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:46.3305880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:46.3315821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:46.3316183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:46.4981059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwugax2u7 2022-09-27T15:56:46.4981935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwugax2u7/_remote_module_non_scriptable.py 2022-09-27T15:56:46.7604099Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:56:46.7614427Z 2022-09-27T15:56:46.7614750Z Running tests... 2022-09-27T15:56:46.7615347Z ---------------------------------------------------------------------- 2022-09-27T15:56:47.0082527Z test_wait_all_workers_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15361 2022-09-27T15:56:47.0109669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15362 2022-09-27T15:56:47.0137294Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15363 2022-09-27T15:56:47.0167277Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15364 2022-09-27T15:56:48.1172554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:48.1173230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:48.1506633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:48.1507355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:48.1662021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:48.1662713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:48.1723175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:48.1723741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:48.1931993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:48.1932524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:48.2056983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:48.2057759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:48.2115182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:48.2115809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:48.2315726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:48.2316456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:48.2794724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8jdhzlj 2022-09-27T15:56:48.2795568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8jdhzlj/_remote_module_non_scriptable.py 2022-09-27T15:56:48.3332541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpampof3i0 2022-09-27T15:56:48.3333805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpampof3i0/_remote_module_non_scriptable.py 2022-09-27T15:56:48.3533601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6woiwal 2022-09-27T15:56:48.3535644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6woiwal/_remote_module_non_scriptable.py 2022-09-27T15:56:48.3613764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4k014hit 2022-09-27T15:56:48.3615452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4k014hit/_remote_module_non_scriptable.py 2022-09-27T15:56:48.5408931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:48.5907136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:56:48.6162343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:56:48.6210359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:48.6482888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:56:48.6585962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:48.6586620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:56:48.6587215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:56:48.6587798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:56:48.6588568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:48.6589402Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:48.6590195Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:49.3927008Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:56:49.3929295Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:56:49.3930859Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:56:49.3931744Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:49.3932557Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:49.3933365Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:49.3934112Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:49.8733391Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #4 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:49.8734102Z [W tensorpipe_agent.cpp:625] RPC agent for worker2 won't send response to request #3 to worker0, as the agent is shutting down 2022-09-27T15:56:49.8828761Z [W tensorpipe_agent.cpp:679] RPC agent for worker0 encountered error when sending response to request #1 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:49.8831677Z [W tensorpipe_agent.cpp:679] RPC agent for worker0 encountered error when sending response to request #1 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:49.8832451Z [W tensorpipe_agent.cpp:679] RPC agent for worker0 encountered error when sending response to request #1 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:50.3741973Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #5 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:50.7245136Z ok (3.963s) 2022-09-27T15:56:50.7245644Z 2022-09-27T15:56:50.7246237Z ---------------------------------------------------------------------- 2022-09-27T15:56:50.7246734Z Ran 1 test in 3.963s 2022-09-27T15:56:50.7246863Z 2022-09-27T15:56:50.7246924Z OK 2022-09-27T15:56:50.7247015Z 2022-09-27T15:56:50.7247109Z Generating XML reports... 2022-09-27T15:56:50.7283523Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155646.xml 2022-09-27T15:56:52.0927294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:52.0927673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:52.0935727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:52.0936103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:52.2537694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphp7yjes8 2022-09-27T15:56:52.2538925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphp7yjes8/_remote_module_non_scriptable.py 2022-09-27T15:56:52.5180774Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:56:52.5190286Z 2022-09-27T15:56:52.5190401Z Running tests... 2022-09-27T15:56:52.5190776Z ---------------------------------------------------------------------- 2022-09-27T15:56:52.7600970Z test_wait_all_workers_twice_dense (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15596 2022-09-27T15:56:52.7626822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15597 2022-09-27T15:56:52.7654116Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15598 2022-09-27T15:56:52.7683156Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15599 2022-09-27T15:56:53.8125931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:53.8126591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:53.8291378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:53.8292063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:53.8489665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:53.8490311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:53.8586608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:53.8587329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:53.8614092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:53.8614749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:53.8658392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:53.8658990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:53.8964578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:53.8965345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:53.8996182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:53.8996854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:53.9762001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjx7f9uc4 2022-09-27T15:56:53.9762805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjx7f9uc4/_remote_module_non_scriptable.py 2022-09-27T15:56:53.9915963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptsbjdri8 2022-09-27T15:56:53.9917096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptsbjdri8/_remote_module_non_scriptable.py 2022-09-27T15:56:54.0217781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi61ah_1s 2022-09-27T15:56:54.0218612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi61ah_1s/_remote_module_non_scriptable.py 2022-09-27T15:56:54.0260078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppv62kxpi 2022-09-27T15:56:54.0261772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppv62kxpi/_remote_module_non_scriptable.py 2022-09-27T15:56:54.2309426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:54.2459449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:54.2771293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:56:54.2844269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:56:54.3124140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:56:54.3226259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:56:54.3226875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:56:54.3227560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:56:54.3228622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:54.3229496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:54.3230263Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:54.3231072Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:56:54.7635334Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:54.7636424Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:56:54.7637252Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:56:54.7638587Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:54.7640388Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T15:56:54.7641324Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:56:55.1743332Z ok (2.655s) 2022-09-27T15:56:55.1743602Z 2022-09-27T15:56:55.1743957Z ---------------------------------------------------------------------- 2022-09-27T15:56:55.1744312Z Ran 1 test in 2.655s 2022-09-27T15:56:55.1744431Z 2022-09-27T15:56:55.1744493Z OK 2022-09-27T15:56:55.1744574Z 2022-09-27T15:56:55.1744672Z Generating XML reports... 2022-09-27T15:56:55.1779104Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155652.xml 2022-09-27T15:56:56.5237903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:56.5238268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:56.5245814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:56.5246381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:56.6811145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwnw166vf 2022-09-27T15:56:56.6812136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwnw166vf/_remote_module_non_scriptable.py 2022-09-27T15:56:56.9406263Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:56:56.9415875Z 2022-09-27T15:56:56.9416260Z Running tests... 2022-09-27T15:56:56.9416662Z ---------------------------------------------------------------------- 2022-09-27T15:56:57.1833028Z test_worker_id (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15831 2022-09-27T15:56:57.1859607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15832 2022-09-27T15:56:57.1887738Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15833 2022-09-27T15:56:57.1917310Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15834 2022-09-27T15:56:58.2846656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:58.2847306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:58.3043733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:58.3044424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:58.3164297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:58.3164786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:58.3201320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:58.3201859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:58.3277198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:56:58.3277701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:56:58.3418462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:58.3419085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:58.3534264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:58.3534867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:58.3642050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:56:58.3642666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:56:58.4463037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptuy7f9xe 2022-09-27T15:56:58.4463797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptuy7f9xe/_remote_module_non_scriptable.py 2022-09-27T15:56:58.4683753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mbdfc95 2022-09-27T15:56:58.4684534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mbdfc95/_remote_module_non_scriptable.py 2022-09-27T15:56:58.4795978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuuoidrjn 2022-09-27T15:56:58.4796668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuuoidrjn/_remote_module_non_scriptable.py 2022-09-27T15:56:58.4902873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgr3b9xa1 2022-09-27T15:56:58.4904291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgr3b9xa1/_remote_module_non_scriptable.py 2022-09-27T15:56:58.7029378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:56:58.7233154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:56:58.7325391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:56:58.7461413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:56:59.3975174Z ok (2.455s) 2022-09-27T15:56:59.3975401Z 2022-09-27T15:56:59.3975709Z ---------------------------------------------------------------------- 2022-09-27T15:56:59.3976022Z Ran 1 test in 2.456s 2022-09-27T15:56:59.3976127Z 2022-09-27T15:56:59.3976190Z OK 2022-09-27T15:56:59.3976284Z 2022-09-27T15:56:59.3976380Z Generating XML reports... 2022-09-27T15:56:59.4011065Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155656.xml 2022-09-27T15:57:00.7499049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:00.7499447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:00.7507327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:00.7507892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:00.9094199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8ro2j_4 2022-09-27T15:57:00.9095519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8ro2j_4/_remote_module_non_scriptable.py 2022-09-27T15:57:01.1703040Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:01.1712444Z 2022-09-27T15:57:01.1712554Z Running tests... 2022-09-27T15:57:01.1713125Z ---------------------------------------------------------------------- 2022-09-27T15:57:01.4135371Z test_worker_info_pickle (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16054 2022-09-27T15:57:01.4162828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16055 2022-09-27T15:57:01.4191061Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16056 2022-09-27T15:57:01.4222193Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16057 2022-09-27T15:57:02.5456862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:02.5457515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:02.5458274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:02.5458836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:02.5523866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:02.5524742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:02.5728797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:02.5729767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:02.5846888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:02.5847416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:02.5850896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:02.5851490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:02.5912983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:02.5913646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:02.6093671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:02.6094370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:02.7125513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7r_9amc2 2022-09-27T15:57:02.7126405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5irg8kp 2022-09-27T15:57:02.7127144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7r_9amc2/_remote_module_non_scriptable.py 2022-09-27T15:57:02.7127861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5irg8kp/_remote_module_non_scriptable.py 2022-09-27T15:57:02.7194471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6p6kcj7 2022-09-27T15:57:02.7195633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6p6kcj7/_remote_module_non_scriptable.py 2022-09-27T15:57:02.7360842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphe__z504 2022-09-27T15:57:02.7362107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphe__z504/_remote_module_non_scriptable.py 2022-09-27T15:57:02.9704585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:02.9723822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:02.9797091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:03.0149734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:03.6277391Z ok (2.456s) 2022-09-27T15:57:03.6277697Z 2022-09-27T15:57:03.6278127Z ---------------------------------------------------------------------- 2022-09-27T15:57:03.6278500Z Ran 1 test in 2.456s 2022-09-27T15:57:03.6278686Z 2022-09-27T15:57:03.6278785Z OK 2022-09-27T15:57:03.6278952Z 2022-09-27T15:57:03.6279071Z Generating XML reports... 2022-09-27T15:57:03.6315071Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155701.xml 2022-09-27T15:57:05.0089261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:05.0089839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:05.0098585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:05.0099194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:05.1676744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt75t4d5i 2022-09-27T15:57:05.1677684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt75t4d5i/_remote_module_non_scriptable.py 2022-09-27T15:57:05.4266188Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:05.4276460Z 2022-09-27T15:57:05.4276911Z Running tests... 2022-09-27T15:57:05.4277518Z ---------------------------------------------------------------------- 2022-09-27T15:57:05.6747503Z test_world_size_one (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16277 2022-09-27T15:57:05.6774250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16278 2022-09-27T15:57:05.6802048Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16279 2022-09-27T15:57:05.6831529Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16280 2022-09-27T15:57:06.8416349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:06.8416981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:06.8495043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:06.8495669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:06.8780435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:06.8781083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:06.8816594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:06.8817245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:06.8857436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:06.8858124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:06.8898261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:06.8898843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:06.9181031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:06.9181707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:06.9263511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:06.9264139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:07.0096566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0omjf6m 2022-09-27T15:57:07.0097373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0omjf6m/_remote_module_non_scriptable.py 2022-09-27T15:57:07.0108784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7m4yn_sw 2022-09-27T15:57:07.0109611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7m4yn_sw/_remote_module_non_scriptable.py 2022-09-27T15:57:07.0446088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpce7_u7t1 2022-09-27T15:57:07.0447652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpce7_u7t1/_remote_module_non_scriptable.py 2022-09-27T15:57:07.0529215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvcwpv4q 2022-09-27T15:57:07.0530484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvcwpv4q/_remote_module_non_scriptable.py 2022-09-27T15:57:07.2662882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:07.2741276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:07.2979947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:07.3084280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:07.6883528Z ok (2.260s) 2022-09-27T15:57:07.6883945Z 2022-09-27T15:57:07.6884253Z ---------------------------------------------------------------------- 2022-09-27T15:57:07.6884556Z Ran 1 test in 2.261s 2022-09-27T15:57:07.6884672Z 2022-09-27T15:57:07.6884732Z OK 2022-09-27T15:57:07.6884823Z 2022-09-27T15:57:07.6884917Z Generating XML reports... 2022-09-27T15:57:07.6919308Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155705.xml 2022-09-27T15:57:09.0413952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:09.0414330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:09.0422419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:09.0422913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:09.2009104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjc2vzci 2022-09-27T15:57:09.2009834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjc2vzci/_remote_module_non_scriptable.py 2022-09-27T15:57:09.4603028Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:09.4612252Z 2022-09-27T15:57:09.4612395Z Running tests... 2022-09-27T15:57:09.4612989Z ---------------------------------------------------------------------- 2022-09-27T15:57:09.7075128Z test_wrong_types (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16374 2022-09-27T15:57:09.7103898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16375 2022-09-27T15:57:09.7132246Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16376 2022-09-27T15:57:09.7161900Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16377 2022-09-27T15:57:10.8136712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:10.8137409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:10.8183114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:10.8183788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:10.8392189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:10.8392867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:10.8405353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:10.8406047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:10.8515716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:10.8516242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:10.8567018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:10.8567666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:10.8774070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:10.8774746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:10.8775539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:10.8776132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:10.9784053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgi3bmq4 2022-09-27T15:57:10.9784895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgi3bmq4/_remote_module_non_scriptable.py 2022-09-27T15:57:10.9829144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplagwdon6 2022-09-27T15:57:10.9831008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplagwdon6/_remote_module_non_scriptable.py 2022-09-27T15:57:11.0037029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_8x8jjx 2022-09-27T15:57:11.0037911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmlwvu64g 2022-09-27T15:57:11.0038636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_8x8jjx/_remote_module_non_scriptable.py 2022-09-27T15:57:11.0041158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmlwvu64g/_remote_module_non_scriptable.py 2022-09-27T15:57:11.2367582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:11.2385183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:11.2607262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:11.2795868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:11.6211944Z ok (2.160s) 2022-09-27T15:57:11.6212185Z 2022-09-27T15:57:11.6212700Z ---------------------------------------------------------------------- 2022-09-27T15:57:11.6212987Z Ran 1 test in 2.160s 2022-09-27T15:57:11.6213107Z 2022-09-27T15:57:11.6213167Z OK 2022-09-27T15:57:11.6213258Z 2022-09-27T15:57:11.6213340Z Generating XML reports... 2022-09-27T15:57:11.6248044Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155709.xml 2022-09-27T15:57:12.9901136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:12.9901527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:12.9909104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:12.9909465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:13.1467876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrmad065 2022-09-27T15:57:13.1469389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrmad065/_remote_module_non_scriptable.py 2022-09-27T15:57:13.4098435Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:13.4108797Z 2022-09-27T15:57:13.4108895Z Running tests... 2022-09-27T15:57:13.4109633Z ---------------------------------------------------------------------- 2022-09-27T15:57:13.6511015Z test_backward_different_dtypes_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16429 2022-09-27T15:57:13.6537493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16430 2022-09-27T15:57:13.6565355Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16431 2022-09-27T15:57:13.6594838Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16432 2022-09-27T15:57:14.8155483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:14.8156184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:14.8342860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:14.8343364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:14.8531297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:14.8531986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:14.8636646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:14.8637286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:14.8721024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:14.8721656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:14.8785379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:14.8785991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:14.9018363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:14.9019038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:14.9158224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:14.9158887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:14.9809053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1uj82flh 2022-09-27T15:57:14.9809843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1uj82flh/_remote_module_non_scriptable.py 2022-09-27T15:57:15.0000106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7kyra6cp 2022-09-27T15:57:15.0000923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7kyra6cp/_remote_module_non_scriptable.py 2022-09-27T15:57:15.0278925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnlf19ejx 2022-09-27T15:57:15.0279714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnlf19ejx/_remote_module_non_scriptable.py 2022-09-27T15:57:15.0412337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjuvllkdy 2022-09-27T15:57:15.0413296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjuvllkdy/_remote_module_non_scriptable.py 2022-09-27T15:57:15.2383800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:15.2598630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:15.2841965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:15.3043319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:15.9652519Z ok (2.554s) 2022-09-27T15:57:15.9652739Z 2022-09-27T15:57:15.9653184Z ---------------------------------------------------------------------- 2022-09-27T15:57:15.9653588Z Ran 1 test in 2.554s 2022-09-27T15:57:15.9653778Z 2022-09-27T15:57:15.9653874Z OK 2022-09-27T15:57:15.9654023Z 2022-09-27T15:57:15.9654159Z Generating XML reports... 2022-09-27T15:57:15.9690326Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155713.xml 2022-09-27T15:57:17.3257887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:17.3258269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:17.3265440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:17.3265841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:17.4846932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpir6iwd2_ 2022-09-27T15:57:17.4847846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpir6iwd2_/_remote_module_non_scriptable.py 2022-09-27T15:57:17.7468439Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:17.7478062Z 2022-09-27T15:57:17.7478208Z Running tests... 2022-09-27T15:57:17.7478803Z ---------------------------------------------------------------------- 2022-09-27T15:57:17.9871150Z test_backward_multiple_round_trips_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16684 2022-09-27T15:57:17.9898968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16685 2022-09-27T15:57:17.9927760Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16686 2022-09-27T15:57:17.9957700Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16687 2022-09-27T15:57:19.1265489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:19.1266102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:19.1615322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:19.1616017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:19.1634321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:19.1635109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:19.1817356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:19.1818019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:19.1985247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:19.1985831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:19.2103529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:19.2104230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:19.2196790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:19.2197420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:19.2453600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:19.2454293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:19.2910027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6dee5x23 2022-09-27T15:57:19.2911295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6dee5x23/_remote_module_non_scriptable.py 2022-09-27T15:57:19.3261646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoudmkbew 2022-09-27T15:57:19.3262385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoudmkbew/_remote_module_non_scriptable.py 2022-09-27T15:57:19.3452659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpze0ovvlc 2022-09-27T15:57:19.3453450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpze0ovvlc/_remote_module_non_scriptable.py 2022-09-27T15:57:19.3709740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvezdgl1 2022-09-27T15:57:19.3710561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvezdgl1/_remote_module_non_scriptable.py 2022-09-27T15:57:19.5510100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:19.5892067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:19.6239031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:19.6354430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:20.4018218Z ok (2.654s) 2022-09-27T15:57:20.4018462Z 2022-09-27T15:57:20.4018977Z ---------------------------------------------------------------------- 2022-09-27T15:57:20.4019388Z Ran 1 test in 2.654s 2022-09-27T15:57:20.4019494Z 2022-09-27T15:57:20.4019553Z OK 2022-09-27T15:57:20.4019645Z 2022-09-27T15:57:20.4019743Z Generating XML reports... 2022-09-27T15:57:20.4055450Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155717.xml 2022-09-27T15:57:21.7692075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:21.7692500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:21.7700917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:21.7701433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:21.9306704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2ax3ko6 2022-09-27T15:57:21.9307858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2ax3ko6/_remote_module_non_scriptable.py 2022-09-27T15:57:22.1939251Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:22.1949518Z 2022-09-27T15:57:22.1950034Z Running tests... 2022-09-27T15:57:22.1950432Z ---------------------------------------------------------------------- 2022-09-27T15:57:22.4413457Z test_backward_no_grad_on_tensor_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16984 2022-09-27T15:57:22.4441377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16985 2022-09-27T15:57:22.4469270Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16986 2022-09-27T15:57:22.4499382Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16987 2022-09-27T15:57:23.5876513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:23.5877172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:23.6137569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:23.6138196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:23.6170972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:23.6171660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:23.6268578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:23.6269092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:23.6516667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:23.6517412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:23.6520687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:23.6521277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:23.6550870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:23.6551866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:23.6893358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:23.6894294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:23.7552683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2kxrdzs 2022-09-27T15:57:23.7553480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2kxrdzs/_remote_module_non_scriptable.py 2022-09-27T15:57:23.7786448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rim20hg 2022-09-27T15:57:23.7787452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rim20hg/_remote_module_non_scriptable.py 2022-09-27T15:57:23.7815481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuhe7h3cl 2022-09-27T15:57:23.7816603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuhe7h3cl/_remote_module_non_scriptable.py 2022-09-27T15:57:23.8162793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxx96d_fa 2022-09-27T15:57:23.8164022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxx96d_fa/_remote_module_non_scriptable.py 2022-09-27T15:57:24.0174254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:24.0379024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:24.0399320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:24.0757626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:24.7560122Z ok (2.560s) 2022-09-27T15:57:24.7560300Z 2022-09-27T15:57:24.7560725Z ---------------------------------------------------------------------- 2022-09-27T15:57:24.7560968Z Ran 1 test in 2.561s 2022-09-27T15:57:24.7561086Z 2022-09-27T15:57:24.7561146Z OK 2022-09-27T15:57:24.7561252Z 2022-09-27T15:57:24.7561382Z Generating XML reports... 2022-09-27T15:57:24.7596904Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155722.xml 2022-09-27T15:57:26.1306687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:26.1307291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:26.1315754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:26.1316373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:26.2884803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ds76gfk 2022-09-27T15:57:26.2885474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ds76gfk/_remote_module_non_scriptable.py 2022-09-27T15:57:26.5504759Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:26.5514747Z 2022-09-27T15:57:26.5515026Z Running tests... 2022-09-27T15:57:26.5515627Z ---------------------------------------------------------------------- 2022-09-27T15:57:26.7994280Z test_backward_rref_multi_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17239 2022-09-27T15:57:26.8021857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17240 2022-09-27T15:57:26.8049685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17241 2022-09-27T15:57:26.8080011Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17242 2022-09-27T15:57:27.9314131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:27.9315210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:27.9345001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:27.9345808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:27.9361609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:27.9362176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:27.9443783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:27.9444275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:27.9699019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:27.9699596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:27.9736097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:27.9736745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:27.9753294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:27.9753951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:27.9827225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:27.9827868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:28.0959344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2wis051e 2022-09-27T15:57:28.0960373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2wis051e/_remote_module_non_scriptable.py 2022-09-27T15:57:28.1002790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp41oynh5w 2022-09-27T15:57:28.1004162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp41oynh5w/_remote_module_non_scriptable.py 2022-09-27T15:57:28.1024462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptem6rm_5 2022-09-27T15:57:28.1026282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptem6rm_5/_remote_module_non_scriptable.py 2022-09-27T15:57:28.1082491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ty4xytp 2022-09-27T15:57:28.1084987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ty4xytp/_remote_module_non_scriptable.py 2022-09-27T15:57:28.3496094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:28.3557148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:28.3576985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:28.3628077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:29.1136722Z ok (2.562s) 2022-09-27T15:57:29.1136898Z 2022-09-27T15:57:29.1137320Z ---------------------------------------------------------------------- 2022-09-27T15:57:29.1137747Z Ran 1 test in 2.562s 2022-09-27T15:57:29.1137953Z 2022-09-27T15:57:29.1138062Z OK 2022-09-27T15:57:29.1138176Z 2022-09-27T15:57:29.1138276Z Generating XML reports... 2022-09-27T15:57:29.1174653Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155726.xml 2022-09-27T15:57:30.4790397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:30.4790829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:30.4798768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:30.4799134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:30.6364794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ichmap6 2022-09-27T15:57:30.6365827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ichmap6/_remote_module_non_scriptable.py 2022-09-27T15:57:30.8971238Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:30.8980874Z 2022-09-27T15:57:30.8981142Z Running tests... 2022-09-27T15:57:30.8981828Z ---------------------------------------------------------------------- 2022-09-27T15:57:31.1398634Z test_backward_rref_nested_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17509 2022-09-27T15:57:31.1426015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17510 2022-09-27T15:57:31.1454486Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17511 2022-09-27T15:57:31.1483996Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17512 2022-09-27T15:57:32.2223159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:32.2223842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:32.2604857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:32.2605522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:32.2704255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:32.2704889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:32.2708554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:32.2709245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:32.2967144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:32.2967706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:32.3101913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:32.3102571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:32.3109099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:32.3109621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:32.3342711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:32.3343339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:32.3891078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4it5w5t 2022-09-27T15:57:32.3891839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4it5w5t/_remote_module_non_scriptable.py 2022-09-27T15:57:32.4379384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25v2qfm3 2022-09-27T15:57:32.4380331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25v2qfm3/_remote_module_non_scriptable.py 2022-09-27T15:57:32.4391518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5b5m09u 2022-09-27T15:57:32.4393778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5b5m09u/_remote_module_non_scriptable.py 2022-09-27T15:57:32.4612137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw87idurk 2022-09-27T15:57:32.4613651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw87idurk/_remote_module_non_scriptable.py 2022-09-27T15:57:32.6472509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:32.6944963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:32.7000867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:32.7195263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:33.4542361Z ok (2.556s) 2022-09-27T15:57:33.4542605Z 2022-09-27T15:57:33.4543128Z ---------------------------------------------------------------------- 2022-09-27T15:57:33.4543408Z Ran 1 test in 2.556s 2022-09-27T15:57:33.4543521Z 2022-09-27T15:57:33.4543581Z OK 2022-09-27T15:57:33.4543669Z 2022-09-27T15:57:33.4543755Z Generating XML reports... 2022-09-27T15:57:33.4579289Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155730.xml 2022-09-27T15:57:34.8258131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:34.8258512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:34.8266224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:34.8266619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:34.9819435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfi9xc00 2022-09-27T15:57:34.9820525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfi9xc00/_remote_module_non_scriptable.py 2022-09-27T15:57:35.2420726Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:35.2429573Z 2022-09-27T15:57:35.2429691Z Running tests... 2022-09-27T15:57:35.2430068Z ---------------------------------------------------------------------- 2022-09-27T15:57:35.4832656Z test_backward_rref_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17785 2022-09-27T15:57:35.4860400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17786 2022-09-27T15:57:35.4889953Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17787 2022-09-27T15:57:35.4919371Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17788 2022-09-27T15:57:36.6284586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:36.6285213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:36.6607485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:36.6608119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:36.6619222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:36.6619917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:36.6860377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:36.6860906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:36.6947920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:36.6960211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:36.6976096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:36.6976917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:36.7240378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:36.7241270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:36.7314207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:36.7315040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:36.7889998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq88dff0j 2022-09-27T15:57:36.7890676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq88dff0j/_remote_module_non_scriptable.py 2022-09-27T15:57:36.8236974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80k0jx2e 2022-09-27T15:57:36.8237840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80k0jx2e/_remote_module_non_scriptable.py 2022-09-27T15:57:36.8513711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9h1wfjw8 2022-09-27T15:57:36.8514879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9h1wfjw8/_remote_module_non_scriptable.py 2022-09-27T15:57:36.8566927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyod8ek41 2022-09-27T15:57:36.8568462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyod8ek41/_remote_module_non_scriptable.py 2022-09-27T15:57:37.0445336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:37.0806091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:37.1111853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:37.1129695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:37.8979132Z ok (2.654s) 2022-09-27T15:57:37.8979338Z 2022-09-27T15:57:37.8979763Z ---------------------------------------------------------------------- 2022-09-27T15:57:37.8980141Z Ran 1 test in 2.655s 2022-09-27T15:57:37.8980319Z 2022-09-27T15:57:37.8980405Z OK 2022-09-27T15:57:37.8980516Z 2022-09-27T15:57:37.8980678Z Generating XML reports... 2022-09-27T15:57:37.9016240Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155735.xml 2022-09-27T15:57:39.2714048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:39.2714473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:39.2723057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:39.2723431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:39.4319484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprz0tjyye 2022-09-27T15:57:39.4320747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprz0tjyye/_remote_module_non_scriptable.py 2022-09-27T15:57:39.6917434Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:39.6927230Z 2022-09-27T15:57:39.6927353Z Running tests... 2022-09-27T15:57:39.6927805Z ---------------------------------------------------------------------- 2022-09-27T15:57:39.9313306Z test_backward_simple_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18064 2022-09-27T15:57:39.9340761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18065 2022-09-27T15:57:39.9368695Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18066 2022-09-27T15:57:39.9399040Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18067 2022-09-27T15:57:41.0879498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:41.0880480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:41.0985919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:41.0986456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:41.1250277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:41.1250901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:41.1345953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:41.1346608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:41.1651352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:41.1651982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:41.1797967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:41.1798629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:41.2040455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:41.2042696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:41.2185682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:41.2186316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:41.2660790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4bk5vb9 2022-09-27T15:57:41.2661568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4bk5vb9/_remote_module_non_scriptable.py 2022-09-27T15:57:41.2701985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdy4wzzsc 2022-09-27T15:57:41.2703422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdy4wzzsc/_remote_module_non_scriptable.py 2022-09-27T15:57:41.3307799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4j7sp3b7 2022-09-27T15:57:41.3309004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4j7sp3b7/_remote_module_non_scriptable.py 2022-09-27T15:57:41.3438796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpako_jgfc 2022-09-27T15:57:41.3439944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpako_jgfc/_remote_module_non_scriptable.py 2022-09-27T15:57:41.5239465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:41.5247196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:41.5879508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:41.6007006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:42.2456328Z ok (2.552s) 2022-09-27T15:57:42.2456563Z 2022-09-27T15:57:42.2457077Z ---------------------------------------------------------------------- 2022-09-27T15:57:42.2457549Z Ran 1 test in 2.553s 2022-09-27T15:57:42.2457741Z 2022-09-27T15:57:42.2457849Z OK 2022-09-27T15:57:42.2458019Z 2022-09-27T15:57:42.2458181Z Generating XML reports... 2022-09-27T15:57:42.2492548Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155739.xml 2022-09-27T15:57:43.6398855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:43.6399315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:43.6406987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:43.6407329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:43.8007279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfuvss6m5 2022-09-27T15:57:43.8008180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfuvss6m5/_remote_module_non_scriptable.py 2022-09-27T15:57:44.0655838Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:44.0665621Z 2022-09-27T15:57:44.0666041Z Running tests... 2022-09-27T15:57:44.0666423Z ---------------------------------------------------------------------- 2022-09-27T15:57:44.3106696Z test_backward_simple_script_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18319 2022-09-27T15:57:44.3133817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18320 2022-09-27T15:57:44.3164295Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18321 2022-09-27T15:57:44.3192444Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18322 2022-09-27T15:57:45.4277097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:45.4277720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:45.4282429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:45.4283015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:45.4414714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:45.4415385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:45.4580940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:45.4581566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:45.4669876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:45.4670606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:45.4676048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:45.4676612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:45.4816095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:45.4816731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:45.4966221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:45.4966906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:45.5950589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphz9gvkpc 2022-09-27T15:57:45.5951422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphz9gvkpc/_remote_module_non_scriptable.py 2022-09-27T15:57:45.5960749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30lga8x2 2022-09-27T15:57:45.5963607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30lga8x2/_remote_module_non_scriptable.py 2022-09-27T15:57:45.6088050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzujr8ywq 2022-09-27T15:57:45.6088704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzujr8ywq/_remote_module_non_scriptable.py 2022-09-27T15:57:45.6226440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfs3isbnk 2022-09-27T15:57:45.6227798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfs3isbnk/_remote_module_non_scriptable.py 2022-09-27T15:57:45.8536093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:45.8557368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:45.8663962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:45.8795294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:46.7252282Z ok (2.658s) 2022-09-27T15:57:46.7252522Z 2022-09-27T15:57:46.7253054Z ---------------------------------------------------------------------- 2022-09-27T15:57:46.7253429Z Ran 1 test in 2.659s 2022-09-27T15:57:46.7253549Z 2022-09-27T15:57:46.7253618Z OK 2022-09-27T15:57:46.7253718Z 2022-09-27T15:57:46.7253811Z Generating XML reports... 2022-09-27T15:57:46.7289041Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155744.xml 2022-09-27T15:57:48.1033552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:48.1033918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:48.1042138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:48.1042496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:48.2590805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeat4k1ss 2022-09-27T15:57:48.2591685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeat4k1ss/_remote_module_non_scriptable.py 2022-09-27T15:57:48.5208199Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:48.5217304Z 2022-09-27T15:57:48.5217392Z Running tests... 2022-09-27T15:57:48.5218227Z ---------------------------------------------------------------------- 2022-09-27T15:57:48.7627019Z test_backward_simple_self_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18574 2022-09-27T15:57:48.7655662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18575 2022-09-27T15:57:48.7684027Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18576 2022-09-27T15:57:48.7714106Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18577 2022-09-27T15:57:49.8540402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:49.8541018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:49.8747888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:49.8748522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:49.8918293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:49.8918941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:49.8991668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:49.8992239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:49.9009700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:49.9010348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:49.9130873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:49.9131503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:49.9369409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:49.9370089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:49.9392112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:49.9392745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:50.0193285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1ftjjgd 2022-09-27T15:57:50.0194111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1ftjjgd/_remote_module_non_scriptable.py 2022-09-27T15:57:50.0398509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp418c8iwk 2022-09-27T15:57:50.0399328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp418c8iwk/_remote_module_non_scriptable.py 2022-09-27T15:57:50.0637088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfnyrfa1 2022-09-27T15:57:50.0637821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfnyrfa1/_remote_module_non_scriptable.py 2022-09-27T15:57:50.0665137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvs7vp4b 2022-09-27T15:57:50.0667147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvs7vp4b/_remote_module_non_scriptable.py 2022-09-27T15:57:50.2764361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:50.2979940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:50.3197423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:50.3272666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:51.0771936Z ok (2.555s) 2022-09-27T15:57:51.0772202Z 2022-09-27T15:57:51.0772512Z ---------------------------------------------------------------------- 2022-09-27T15:57:51.0772762Z Ran 1 test in 2.555s 2022-09-27T15:57:51.0772878Z 2022-09-27T15:57:51.0772937Z OK 2022-09-27T15:57:51.0773014Z 2022-09-27T15:57:51.0773112Z Generating XML reports... 2022-09-27T15:57:51.0808195Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155748.xml 2022-09-27T15:57:52.4650348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:52.4650735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:52.4658377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:52.4658742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:52.6248284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpydix97st 2022-09-27T15:57:52.6249787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpydix97st/_remote_module_non_scriptable.py 2022-09-27T15:57:52.8866876Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:52.8876405Z 2022-09-27T15:57:52.8876792Z Running tests... 2022-09-27T15:57:52.8877191Z ---------------------------------------------------------------------- 2022-09-27T15:57:53.1291725Z test_backward_simple_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18829 2022-09-27T15:57:53.1319070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18830 2022-09-27T15:57:53.1348287Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18831 2022-09-27T15:57:53.1380356Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18832 2022-09-27T15:57:54.2446472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:54.2447075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:54.2799420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:54.2800011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:54.2829019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:54.2829632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:54.2839457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:54.2840072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:54.3054571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:54.3055253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:54.3206014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:54.3206591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:54.3218867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:54.3219464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:54.3413745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:54.3414384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:54.4070041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgqayt3j 2022-09-27T15:57:54.4070762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgqayt3j/_remote_module_non_scriptable.py 2022-09-27T15:57:54.4467107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8r4o_ip3 2022-09-27T15:57:54.4468064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8r4o_ip3/_remote_module_non_scriptable.py 2022-09-27T15:57:54.4486438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpihtafe6b 2022-09-27T15:57:54.4488056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpihtafe6b/_remote_module_non_scriptable.py 2022-09-27T15:57:54.4681044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp965bkkjx 2022-09-27T15:57:54.4681977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp965bkkjx/_remote_module_non_scriptable.py 2022-09-27T15:57:54.6648693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:54.7049155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:54.7073359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:54.7455631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:55.4438868Z ok (2.556s) 2022-09-27T15:57:55.4439059Z 2022-09-27T15:57:55.4439371Z ---------------------------------------------------------------------- 2022-09-27T15:57:55.4439637Z Ran 1 test in 2.556s 2022-09-27T15:57:55.4439753Z 2022-09-27T15:57:55.4439815Z OK 2022-09-27T15:57:55.4440127Z 2022-09-27T15:57:55.4440227Z Generating XML reports... 2022-09-27T15:57:55.4474760Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155752.xml 2022-09-27T15:57:56.8144351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:56.8144961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:56.8153568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:56.8154199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:56.9742792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpliw_mdhu 2022-09-27T15:57:56.9744299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpliw_mdhu/_remote_module_non_scriptable.py 2022-09-27T15:57:57.2385490Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:57:57.2395913Z 2022-09-27T15:57:57.2396342Z Running tests... 2022-09-27T15:57:57.2396746Z ---------------------------------------------------------------------- 2022-09-27T15:57:57.4855106Z test_backwards_nested_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19084 2022-09-27T15:57:57.4883733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19085 2022-09-27T15:57:57.4912319Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19086 2022-09-27T15:57:57.4942204Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19087 2022-09-27T15:57:58.6460464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:58.6461111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:58.6727095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:58.6727716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:58.6819795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:58.6820413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:58.6833541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:58.6834150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:58.6959779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:57:58.6960465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:57:58.7114780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:58.7115355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:58.7207760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:58.7208424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:58.7333089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:57:58.7333761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:57:58.8138148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02iry80j 2022-09-27T15:57:58.8138854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02iry80j/_remote_module_non_scriptable.py 2022-09-27T15:57:58.8410869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaxlcm1sd 2022-09-27T15:57:58.8411678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaxlcm1sd/_remote_module_non_scriptable.py 2022-09-27T15:57:58.8489484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9s6xx6_ 2022-09-27T15:57:58.8490488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9s6xx6_/_remote_module_non_scriptable.py 2022-09-27T15:57:58.8769874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujho7x3x 2022-09-27T15:57:58.8770648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujho7x3x/_remote_module_non_scriptable.py 2022-09-27T15:57:59.0733105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:57:59.0987376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:57:59.1041769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:57:59.1334154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:57:59.8001318Z ok (2.560s) 2022-09-27T15:57:59.8001583Z 2022-09-27T15:57:59.8002075Z ---------------------------------------------------------------------- 2022-09-27T15:57:59.8002542Z Ran 1 test in 2.560s 2022-09-27T15:57:59.8002746Z 2022-09-27T15:57:59.8002838Z OK 2022-09-27T15:57:59.8003012Z 2022-09-27T15:57:59.8003177Z Generating XML reports... 2022-09-27T15:57:59.8036995Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155757.xml 2022-09-27T15:58:01.1663814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:01.1664199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:01.1671887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:01.1672258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:01.3268993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2lfuev5d 2022-09-27T15:58:01.3269938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2lfuev5d/_remote_module_non_scriptable.py 2022-09-27T15:58:01.5860685Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:58:01.5869952Z 2022-09-27T15:58:01.5870044Z Running tests... 2022-09-27T15:58:01.5870786Z ---------------------------------------------------------------------- 2022-09-27T15:58:01.8316336Z test_context_cleanup_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19339 2022-09-27T15:58:01.8343296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19340 2022-09-27T15:58:01.8371620Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19341 2022-09-27T15:58:01.8401897Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19342 2022-09-27T15:58:02.9808306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:02.9809009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:03.0023431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:03.0024091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:03.0168357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:03.0168999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:03.0385641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:03.0386304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:03.0451807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:03.0452394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:03.0477548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:03.0478193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:03.0821016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:03.0821751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:03.0851238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:03.0851827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:03.1442882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjp2ltk5d 2022-09-27T15:58:03.1443704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjp2ltk5d/_remote_module_non_scriptable.py 2022-09-27T15:58:03.1649965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkw2cc2be 2022-09-27T15:58:03.1650644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkw2cc2be/_remote_module_non_scriptable.py 2022-09-27T15:58:03.2077243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvtbr1q7j 2022-09-27T15:58:03.2078645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvtbr1q7j/_remote_module_non_scriptable.py 2022-09-27T15:58:03.2104509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiatun_z7 2022-09-27T15:58:03.2105832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiatun_z7/_remote_module_non_scriptable.py 2022-09-27T15:58:03.3997922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:58:03.4219887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:58:03.4643161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:58:03.4650966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:58:03.8153269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:58:03.8253429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:58:03.8361596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:58:03.8362489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:58:03.8366330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:03.8367401Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:03.8368536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:03.8369457Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:04.3463522Z ok (2.759s) 2022-09-27T15:58:04.3463770Z 2022-09-27T15:58:04.3464238Z ---------------------------------------------------------------------- 2022-09-27T15:58:04.3464691Z Ran 1 test in 2.759s 2022-09-27T15:58:04.3464885Z 2022-09-27T15:58:04.3464976Z OK 2022-09-27T15:58:04.3465070Z 2022-09-27T15:58:04.3465378Z Generating XML reports... 2022-09-27T15:58:04.3500797Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155801.xml 2022-09-27T15:58:05.7310614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:05.7310989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:05.7318851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:05.7319339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:05.8856552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaaoui_vt 2022-09-27T15:58:05.8857693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaaoui_vt/_remote_module_non_scriptable.py 2022-09-27T15:58:06.1463039Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:58:06.1472736Z 2022-09-27T15:58:06.1472856Z Running tests... 2022-09-27T15:58:06.1473467Z ---------------------------------------------------------------------- 2022-09-27T15:58:06.3894455Z test_context_cleanup_tensor_no_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19586 2022-09-27T15:58:06.3922502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19587 2022-09-27T15:58:06.3950370Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19588 2022-09-27T15:58:06.3979541Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19589 2022-09-27T15:58:07.4661760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:07.4662414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:07.4772098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:07.4772776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:07.4956984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:07.4957497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:07.5019740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:07.5020403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:07.5142329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:07.5142949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:07.5306770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:07.5307427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:07.5358918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:07.5359551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:07.5688476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:07.5689115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:07.6284411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zwl0w32 2022-09-27T15:58:07.6285641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zwl0w32/_remote_module_non_scriptable.py 2022-09-27T15:58:07.6402876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphoceu2uy 2022-09-27T15:58:07.6403556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphoceu2uy/_remote_module_non_scriptable.py 2022-09-27T15:58:07.6572138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqm6h7k05 2022-09-27T15:58:07.6573085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqm6h7k05/_remote_module_non_scriptable.py 2022-09-27T15:58:07.6953156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpovkkgxss 2022-09-27T15:58:07.6953882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpovkkgxss/_remote_module_non_scriptable.py 2022-09-27T15:58:07.8868285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:58:07.8973912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:58:07.9162567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:58:07.9536694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:58:08.3006277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:58:08.3106661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:58:08.3110715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:58:08.3111452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:58:08.3112293Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:08.3112849Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:08.3209219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:08.3210014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:08.9041122Z ok (2.756s) 2022-09-27T15:58:08.9041409Z 2022-09-27T15:58:08.9041869Z ---------------------------------------------------------------------- 2022-09-27T15:58:08.9042114Z Ran 1 test in 2.757s 2022-09-27T15:58:08.9042233Z 2022-09-27T15:58:08.9042296Z OK 2022-09-27T15:58:08.9042390Z 2022-09-27T15:58:08.9042488Z Generating XML reports... 2022-09-27T15:58:08.9079207Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155806.xml 2022-09-27T15:58:10.3245873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:10.3246255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:10.3254514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:10.3254917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:10.4842912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcplzty6f 2022-09-27T15:58:10.4843405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcplzty6f/_remote_module_non_scriptable.py 2022-09-27T15:58:10.7494108Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:58:10.7503034Z 2022-09-27T15:58:10.7503156Z Running tests... 2022-09-27T15:58:10.7503763Z ---------------------------------------------------------------------- 2022-09-27T15:58:11.0019673Z test_context_cleanup_tensor_with_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19833 2022-09-27T15:58:11.0049988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19834 2022-09-27T15:58:11.0079408Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19835 2022-09-27T15:58:11.0109915Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19836 2022-09-27T15:58:12.1385241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:12.1385828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:12.1503633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:12.1504240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:12.1632312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:12.1632845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:12.1768141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:12.1768791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:12.1815824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:12.1816428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:12.1892792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:12.1893416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:12.2019062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:12.2019670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:12.2285483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:12.2286153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:12.3048824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4fd_4zz 2022-09-27T15:58:12.3049610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4fd_4zz/_remote_module_non_scriptable.py 2022-09-27T15:58:12.3184141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20d54pyy 2022-09-27T15:58:12.3184962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20d54pyy/_remote_module_non_scriptable.py 2022-09-27T15:58:12.3324675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtu4mzjd 2022-09-27T15:58:12.3326322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtu4mzjd/_remote_module_non_scriptable.py 2022-09-27T15:58:12.3600920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporewuyj7 2022-09-27T15:58:12.3602127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporewuyj7/_remote_module_non_scriptable.py 2022-09-27T15:58:12.5605653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:58:12.5729973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:58:12.5895161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:58:12.6171620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:58:12.8834240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:58:12.8937472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:58:12.8939419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:58:12.8940607Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:12.8943142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:12.8944705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:12.8946538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:58:12.8947614Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:13.4168768Z ok (2.666s) 2022-09-27T15:58:13.4168976Z 2022-09-27T15:58:13.4169433Z ---------------------------------------------------------------------- 2022-09-27T15:58:13.4169821Z Ran 1 test in 2.666s 2022-09-27T15:58:13.4170024Z 2022-09-27T15:58:13.4170132Z OK 2022-09-27T15:58:13.4170283Z 2022-09-27T15:58:13.4170425Z Generating XML reports... 2022-09-27T15:58:13.4206261Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155810.xml 2022-09-27T15:58:14.7977698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:14.7978076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:14.7986462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:14.7986827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:14.9588923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjbkov8i 2022-09-27T15:58:14.9589836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjbkov8i/_remote_module_non_scriptable.py 2022-09-27T15:58:15.2241542Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:58:15.2250821Z 2022-09-27T15:58:15.2250943Z Running tests... 2022-09-27T15:58:15.2251505Z ---------------------------------------------------------------------- 2022-09-27T15:58:15.4732369Z test_embedding_bag_with_no_grad_tensors (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20080 2022-09-27T15:58:15.4760518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20081 2022-09-27T15:58:15.4788160Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20082 2022-09-27T15:58:15.4817769Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20083 2022-09-27T15:58:16.6049322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:16.6049978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:16.6128985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:16.6129696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:16.6138236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:16.6138885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:16.6222080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:16.6222603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:16.6437302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:16.6438189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:16.6517432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:16.6518362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:16.6528151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:16.6528809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:16.6606457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:16.6607119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:16.7706690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxk30fkt 2022-09-27T15:58:16.7707510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxk30fkt/_remote_module_non_scriptable.py 2022-09-27T15:58:16.7795587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjf_0eccs 2022-09-27T15:58:16.7796701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpetl7ldd0 2022-09-27T15:58:16.7797358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjf_0eccs/_remote_module_non_scriptable.py 2022-09-27T15:58:16.7798512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpetl7ldd0/_remote_module_non_scriptable.py 2022-09-27T15:58:16.7883754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6jrydrq 2022-09-27T15:58:16.7884776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6jrydrq/_remote_module_non_scriptable.py 2022-09-27T15:58:17.0257253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:58:17.0360655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:58:17.0361379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:58:17.0469491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:58:17.7873861Z ok (2.562s) 2022-09-27T15:58:17.7874113Z 2022-09-27T15:58:17.7874707Z ---------------------------------------------------------------------- 2022-09-27T15:58:17.7875000Z Ran 1 test in 2.562s 2022-09-27T15:58:17.7875124Z 2022-09-27T15:58:17.7875185Z OK 2022-09-27T15:58:17.7875275Z 2022-09-27T15:58:17.7875372Z Generating XML reports... 2022-09-27T15:58:17.7913010Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155815.xml 2022-09-27T15:58:19.1495287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:19.1495662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:19.1504221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:19.1504611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:19.3099426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ba0gqi4 2022-09-27T15:58:19.3100162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ba0gqi4/_remote_module_non_scriptable.py 2022-09-27T15:58:19.5713302Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:58:19.5722766Z 2022-09-27T15:58:19.5723129Z Running tests... 2022-09-27T15:58:19.5723463Z ---------------------------------------------------------------------- 2022-09-27T15:58:19.8191472Z test_graph_for_builtin_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20347 2022-09-27T15:58:19.8217906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20348 2022-09-27T15:58:19.8245700Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20349 2022-09-27T15:58:19.8275760Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20350 2022-09-27T15:58:20.9705959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:20.9706691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:20.9943201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:20.9943704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:20.9978486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:20.9979186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:21.0067943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:21.0068475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:21.0079346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:21.0079880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:21.0326201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:21.0326951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:21.0365508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:21.0366156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:21.0439836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:21.0440478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:21.1348754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfgtyw7q 2022-09-27T15:58:21.1349483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfgtyw7q/_remote_module_non_scriptable.py 2022-09-27T15:58:21.1593773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmponp7mpw8 2022-09-27T15:58:21.1595252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmponp7mpw8/_remote_module_non_scriptable.py 2022-09-27T15:58:21.1629250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8m18_p1v 2022-09-27T15:58:21.1631003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8m18_p1v/_remote_module_non_scriptable.py 2022-09-27T15:58:21.1702954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgdsmulq9 2022-09-27T15:58:21.1704369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgdsmulq9/_remote_module_non_scriptable.py 2022-09-27T15:58:21.3925548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:58:21.4183414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:58:21.4210203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:58:21.4263549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:58:21.6668013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:58:21.6868327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:58:21.6869863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:58:21.6872460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:58:21.6873850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:21.6971538Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:21.6972438Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:21.6973257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:22.2334604Z ok (2.661s) 2022-09-27T15:58:22.2334839Z 2022-09-27T15:58:22.2335314Z ---------------------------------------------------------------------- 2022-09-27T15:58:22.2335707Z Ran 1 test in 2.661s 2022-09-27T15:58:22.2335891Z 2022-09-27T15:58:22.2336000Z OK 2022-09-27T15:58:22.2336146Z 2022-09-27T15:58:22.2336291Z Generating XML reports... 2022-09-27T15:58:22.2372970Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155819.xml 2022-09-27T15:58:23.6223981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:23.6224403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:23.6232407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:23.6232853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:23.7821794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0wx158m 2022-09-27T15:58:23.7822638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0wx158m/_remote_module_non_scriptable.py 2022-09-27T15:58:24.0437844Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:58:24.0447302Z 2022-09-27T15:58:24.0447659Z Running tests... 2022-09-27T15:58:24.0448064Z ---------------------------------------------------------------------- 2022-09-27T15:58:24.2870857Z test_graph_for_builtin_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20594 2022-09-27T15:58:24.2898557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20595 2022-09-27T15:58:24.2927015Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20596 2022-09-27T15:58:24.2957237Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20597 2022-09-27T15:58:25.3747051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:25.4051969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:25.4052917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:25.4053561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:25.4054346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:25.4054959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:25.4116253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:25.4116803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:25.4418610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:25.4419248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:25.4429197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:25.4430007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:25.4547730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:25.4548340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:25.4905140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:25.4905810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:25.5392071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb13zuyup 2022-09-27T15:58:25.5392922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb13zuyup/_remote_module_non_scriptable.py 2022-09-27T15:58:25.5685331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39ix67kk 2022-09-27T15:58:25.5686140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39ix67kk/_remote_module_non_scriptable.py 2022-09-27T15:58:25.5693864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56_tumbz 2022-09-27T15:58:25.5696370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56_tumbz/_remote_module_non_scriptable.py 2022-09-27T15:58:25.6169061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitkeb6df 2022-09-27T15:58:25.6169898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitkeb6df/_remote_module_non_scriptable.py 2022-09-27T15:58:25.8022705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:58:25.8289587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:58:25.8320668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:58:25.8960422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:58:26.1689768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:58:26.1690627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:58:26.1691398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:58:26.1692979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:58:26.1695071Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:26.1793793Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:26.1795136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:26.1796294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:26.8019710Z ok (2.757s) 2022-09-27T15:58:26.8019900Z 2022-09-27T15:58:26.8020232Z ---------------------------------------------------------------------- 2022-09-27T15:58:26.8020515Z Ran 1 test in 2.757s 2022-09-27T15:58:26.8020621Z 2022-09-27T15:58:26.8020685Z OK 2022-09-27T15:58:26.8020776Z 2022-09-27T15:58:26.8020876Z Generating XML reports... 2022-09-27T15:58:26.8055979Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155824.xml 2022-09-27T15:58:28.1983039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:28.1983649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:28.1992550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:28.1993144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:28.3610492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvi36ryx8 2022-09-27T15:58:28.3611262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvi36ryx8/_remote_module_non_scriptable.py 2022-09-27T15:58:28.6300702Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:58:28.6310125Z 2022-09-27T15:58:28.6310269Z Running tests... 2022-09-27T15:58:28.6310880Z ---------------------------------------------------------------------- 2022-09-27T15:58:28.8814417Z test_graph_for_py_nested_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20841 2022-09-27T15:58:28.8842234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20842 2022-09-27T15:58:28.8871187Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20843 2022-09-27T15:58:28.8908274Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20844 2022-09-27T15:58:30.0036229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:30.0036890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:30.0078360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:30.0079033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:30.0287497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:30.0288193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:30.0394376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:30.0395129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:30.0413395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:30.0413918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:30.0462904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:30.0463428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:30.0670535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:30.0671251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:30.0764802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:30.0765496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:30.1674291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8um6s6_i 2022-09-27T15:58:30.1675175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8um6s6_i/_remote_module_non_scriptable.py 2022-09-27T15:58:30.1726591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpktoju82t 2022-09-27T15:58:30.1728516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpktoju82t/_remote_module_non_scriptable.py 2022-09-27T15:58:30.1935005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2rrpq4y 2022-09-27T15:58:30.1935820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2rrpq4y/_remote_module_non_scriptable.py 2022-09-27T15:58:30.2017166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwaax9zg 2022-09-27T15:58:30.2018156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwaax9zg/_remote_module_non_scriptable.py 2022-09-27T15:58:30.4253893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:58:30.4317089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:58:30.4512187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:58:30.4578366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:58:30.7151259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:58:30.7251929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:58:30.7253598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:58:30.7254385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:58:30.7255659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:30.7256800Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:30.7257917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:30.7259058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:31.2970513Z ok (2.666s) 2022-09-27T15:58:31.2970767Z 2022-09-27T15:58:31.2971273Z ---------------------------------------------------------------------- 2022-09-27T15:58:31.2971688Z Ran 1 test in 2.666s 2022-09-27T15:58:31.2971809Z 2022-09-27T15:58:31.2971870Z OK 2022-09-27T15:58:31.2971963Z 2022-09-27T15:58:31.2972066Z Generating XML reports... 2022-09-27T15:58:31.3006223Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155828.xml 2022-09-27T15:58:32.7040751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:32.7041146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:32.7048897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:32.7049267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:32.8630910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj00m9j5o 2022-09-27T15:58:32.8632489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj00m9j5o/_remote_module_non_scriptable.py 2022-09-27T15:58:33.1232277Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:58:33.1241913Z 2022-09-27T15:58:33.1242022Z Running tests... 2022-09-27T15:58:33.1242602Z ---------------------------------------------------------------------- 2022-09-27T15:58:33.3672307Z test_graph_for_py_nested_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21088 2022-09-27T15:58:33.3699289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21089 2022-09-27T15:58:33.3727171Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21090 2022-09-27T15:58:33.3756222Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21091 2022-09-27T15:58:34.4157214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:34.4157845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:34.4274402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:34.4275108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:34.4398706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:34.4399363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:34.4529722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:34.4530331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:34.4645410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:34.4646113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:34.4762224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:34.4762899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:34.5013412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:34.5014021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:34.5354765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:34.5355450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:34.5791684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdm0phjdo 2022-09-27T15:58:34.5792594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdm0phjdo/_remote_module_non_scriptable.py 2022-09-27T15:58:34.5912264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3kpf01bw 2022-09-27T15:58:34.5915440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3kpf01bw/_remote_module_non_scriptable.py 2022-09-27T15:58:34.6026385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk41kix0e 2022-09-27T15:58:34.6027544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk41kix0e/_remote_module_non_scriptable.py 2022-09-27T15:58:34.6608878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmoavfym1 2022-09-27T15:58:34.6610625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmoavfym1/_remote_module_non_scriptable.py 2022-09-27T15:58:34.8340804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:58:34.8464288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:58:34.8580014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:58:34.9172887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:58:35.2089725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:58:35.2090583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:58:35.2200250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:58:35.2201141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:58:35.2202405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:35.2203803Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:35.2204929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:35.2240534Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:35.8817635Z ok (2.757s) 2022-09-27T15:58:35.8817893Z 2022-09-27T15:58:35.8818387Z ---------------------------------------------------------------------- 2022-09-27T15:58:35.8818824Z Ran 1 test in 2.757s 2022-09-27T15:58:35.8819027Z 2022-09-27T15:58:35.8819141Z OK 2022-09-27T15:58:35.8832040Z 2022-09-27T15:58:35.8832397Z Generating XML reports... 2022-09-27T15:58:35.8854755Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155833.xml 2022-09-27T15:58:37.2506894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:37.2507275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:37.2515667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:37.2516059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:37.4083860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdr00f8dr 2022-09-27T15:58:37.4084769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdr00f8dr/_remote_module_non_scriptable.py 2022-09-27T15:58:37.6699637Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:58:37.6709301Z 2022-09-27T15:58:37.6709443Z Running tests... 2022-09-27T15:58:37.6709930Z ---------------------------------------------------------------------- 2022-09-27T15:58:37.9120034Z test_graph_for_py_nested_remote_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21335 2022-09-27T15:58:37.9147735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21336 2022-09-27T15:58:37.9175285Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21337 2022-09-27T15:58:37.9206331Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21338 2022-09-27T15:58:39.0145931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:39.0146552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:39.0323545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:39.0498789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:39.0499708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:39.0500283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:39.0516807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:39.0517442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:39.0603448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:39.0603985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:39.0711013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:39.0711638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:39.0883467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:39.0884412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:39.0983670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:39.0984607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:39.1801951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53b3ph34 2022-09-27T15:58:39.1802995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53b3ph34/_remote_module_non_scriptable.py 2022-09-27T15:58:39.2002494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpipteumel 2022-09-27T15:58:39.2003293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpipteumel/_remote_module_non_scriptable.py 2022-09-27T15:58:39.2175031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsp0ai7l5 2022-09-27T15:58:39.2175861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsp0ai7l5/_remote_module_non_scriptable.py 2022-09-27T15:58:39.2440358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79vjfnjz 2022-09-27T15:58:39.2441255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79vjfnjz/_remote_module_non_scriptable.py 2022-09-27T15:58:39.4388500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:58:39.4560780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:58:39.4742831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:58:39.5026493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:58:39.7662227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:58:39.7764640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:58:39.7873518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:58:39.7874378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:39.7875075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:58:39.7875751Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:39.7876438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:39.7877142Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:40.4266449Z ok (2.755s) 2022-09-27T15:58:40.4266684Z 2022-09-27T15:58:40.4267205Z ---------------------------------------------------------------------- 2022-09-27T15:58:40.4267657Z Ran 1 test in 2.756s 2022-09-27T15:58:40.4267877Z 2022-09-27T15:58:40.4267987Z OK 2022-09-27T15:58:40.4268154Z 2022-09-27T15:58:40.4268320Z Generating XML reports... 2022-09-27T15:58:40.4303676Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155837.xml 2022-09-27T15:58:41.8261471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:41.8261859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:41.8269487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:41.8270000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:41.9823831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dwb5wfd 2022-09-27T15:58:41.9824741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dwb5wfd/_remote_module_non_scriptable.py 2022-09-27T15:58:42.2431771Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:58:42.2440845Z 2022-09-27T15:58:42.2440979Z Running tests... 2022-09-27T15:58:42.2441559Z ---------------------------------------------------------------------- 2022-09-27T15:58:42.4901261Z test_graph_for_py_nested_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21582 2022-09-27T15:58:42.4929140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21583 2022-09-27T15:58:42.4957188Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21584 2022-09-27T15:58:42.4986604Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21585 2022-09-27T15:58:43.5674396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:43.5675233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:43.6012645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:43.6013298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:43.6104334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:43.6104949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:43.6191020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:43.6191717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:43.6240250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:43.6240867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:43.6469649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:43.6470286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:43.6559172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:43.6559840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:43.6606028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:43.6606632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:43.7276069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5f8_jbnn 2022-09-27T15:58:43.7276890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5f8_jbnn/_remote_module_non_scriptable.py 2022-09-27T15:58:43.7745807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpetiir74r 2022-09-27T15:58:43.7746602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpetiir74r/_remote_module_non_scriptable.py 2022-09-27T15:58:43.7812662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu4n5pcgc 2022-09-27T15:58:43.7814167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu4n5pcgc/_remote_module_non_scriptable.py 2022-09-27T15:58:43.7863693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkk56jqgm 2022-09-27T15:58:43.7865250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkk56jqgm/_remote_module_non_scriptable.py 2022-09-27T15:58:43.9810692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:58:44.0298874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:58:44.0341113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:58:44.0423653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:58:44.2488006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:58:44.2586545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:58:44.2688193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:58:44.2689534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:58:44.2690828Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:44.2691985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:44.2693160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:44.2699283Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:44.9047345Z ok (2.660s) 2022-09-27T15:58:44.9047582Z 2022-09-27T15:58:44.9048037Z ---------------------------------------------------------------------- 2022-09-27T15:58:44.9048436Z Ran 1 test in 2.661s 2022-09-27T15:58:44.9048614Z 2022-09-27T15:58:44.9048691Z OK 2022-09-27T15:58:44.9048831Z 2022-09-27T15:58:44.9048974Z Generating XML reports... 2022-09-27T15:58:44.9084910Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155842.xml 2022-09-27T15:58:46.2869692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:46.2870338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:46.2879105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:46.2879727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:46.4500382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf50d9zme 2022-09-27T15:58:46.4501501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf50d9zme/_remote_module_non_scriptable.py 2022-09-27T15:58:46.7097010Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:58:46.7106387Z 2022-09-27T15:58:46.7106525Z Running tests... 2022-09-27T15:58:46.7107125Z ---------------------------------------------------------------------- 2022-09-27T15:58:46.9536862Z test_graph_for_python_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21829 2022-09-27T15:58:46.9563250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21830 2022-09-27T15:58:46.9590864Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21831 2022-09-27T15:58:46.9621175Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21832 2022-09-27T15:58:48.1283402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:48.1283993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:48.1291833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:48.1292472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:48.1665399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:48.1666121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:48.1669343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:48.1669964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:48.1718343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:48.1718953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:48.1979607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:48.1980298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:48.2078400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:48.2078918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:48.2358374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:48.2359031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:48.2937281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5771zau 2022-09-27T15:58:48.2938052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5771zau/_remote_module_non_scriptable.py 2022-09-27T15:58:48.2947217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69f3qr2w 2022-09-27T15:58:48.2949242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69f3qr2w/_remote_module_non_scriptable.py 2022-09-27T15:58:48.3333079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rqvieui 2022-09-27T15:58:48.3333812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rqvieui/_remote_module_non_scriptable.py 2022-09-27T15:58:48.3625266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5ketsqb 2022-09-27T15:58:48.3626182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5ketsqb/_remote_module_non_scriptable.py 2022-09-27T15:58:48.5514397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:58:48.5732993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:58:48.5897364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:58:48.6241640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:58:48.8809112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:58:48.8909439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:58:48.9011146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:58:48.9012312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:58:48.9014392Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:48.9015559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:48.9016702Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:48.9017841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:49.5684618Z ok (2.857s) 2022-09-27T15:58:49.5684880Z 2022-09-27T15:58:49.5685354Z ---------------------------------------------------------------------- 2022-09-27T15:58:49.5685918Z Ran 1 test in 2.858s 2022-09-27T15:58:49.5686127Z 2022-09-27T15:58:49.5686227Z OK 2022-09-27T15:58:49.5686355Z 2022-09-27T15:58:49.5686450Z Generating XML reports... 2022-09-27T15:58:49.5720576Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155846.xml 2022-09-27T15:58:50.9644111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:50.9644512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:50.9652285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:50.9652642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:51.1229357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsc5l2o8s 2022-09-27T15:58:51.1230618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsc5l2o8s/_remote_module_non_scriptable.py 2022-09-27T15:58:51.3839792Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:58:51.3849063Z 2022-09-27T15:58:51.3849441Z Running tests... 2022-09-27T15:58:51.3849831Z ---------------------------------------------------------------------- 2022-09-27T15:58:51.6338819Z test_graph_for_python_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22076 2022-09-27T15:58:51.6366050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22077 2022-09-27T15:58:51.6393441Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22078 2022-09-27T15:58:51.6422325Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22079 2022-09-27T15:58:52.7705047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:52.7705722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:52.8072755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:52.8073465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:52.8497970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:52.8498599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:52.8531495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:52.8532158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:52.8543481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:52.8544121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:52.8874443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:52.8875303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:52.8913261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:52.8913895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:52.8920299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:52.8920896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:52.9348380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx12awsdj 2022-09-27T15:58:52.9349311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx12awsdj/_remote_module_non_scriptable.py 2022-09-27T15:58:53.0135155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6a2zbv_s 2022-09-27T15:58:53.0135980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6a2zbv_s/_remote_module_non_scriptable.py 2022-09-27T15:58:53.0171862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zr2r5iy 2022-09-27T15:58:53.0173217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zr2r5iy/_remote_module_non_scriptable.py 2022-09-27T15:58:53.0178970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qad2eso 2022-09-27T15:58:53.0181079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qad2eso/_remote_module_non_scriptable.py 2022-09-27T15:58:53.1916591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:58:53.2713834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:58:53.2719675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:58:53.2725529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:58:53.5424429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:58:53.5523358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:58:53.5627489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:58:53.5628787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:53.5629647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:58:53.5630704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:53.5632311Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:53.5639352Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:58:54.1483554Z ok (2.763s) 2022-09-27T15:58:54.1483794Z 2022-09-27T15:58:54.1484308Z ---------------------------------------------------------------------- 2022-09-27T15:58:54.1484732Z Ran 1 test in 2.763s 2022-09-27T15:58:54.1484855Z 2022-09-27T15:58:54.1484917Z OK 2022-09-27T15:58:54.1485008Z 2022-09-27T15:58:54.1485100Z Generating XML reports... 2022-09-27T15:58:54.1522287Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155851.xml 2022-09-27T15:58:55.5608183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:55.5608594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:55.5616319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:55.5616703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:55.7228639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3mlsc7p 2022-09-27T15:58:55.7229633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3mlsc7p/_remote_module_non_scriptable.py 2022-09-27T15:58:55.9926827Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:58:55.9936007Z 2022-09-27T15:58:55.9936122Z Running tests... 2022-09-27T15:58:55.9936940Z ---------------------------------------------------------------------- 2022-09-27T15:58:56.2417142Z test_mixed_requires_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22323 2022-09-27T15:58:56.2444733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22324 2022-09-27T15:58:56.2472827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22325 2022-09-27T15:58:56.2502261Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22326 2022-09-27T15:58:57.4042619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:57.4043203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:57.4051623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:57.4052302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:57.4316135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:57.4316781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:57.4326845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:57.4327440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:57.4424166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:57.4424892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:57.4437381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:57.4437941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:57.4694904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:57.4695580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:57.4703800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:57.4704451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:58:57.5685496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmxav5hn6 2022-09-27T15:58:57.5686210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmxav5hn6/_remote_module_non_scriptable.py 2022-09-27T15:58:57.5692469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplx1xmpgj 2022-09-27T15:58:57.5694196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplx1xmpgj/_remote_module_non_scriptable.py 2022-09-27T15:58:57.5961578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jp924e7 2022-09-27T15:58:57.5962565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jp924e7/_remote_module_non_scriptable.py 2022-09-27T15:58:57.5964500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvx0gw5gq 2022-09-27T15:58:57.5967345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvx0gw5gq/_remote_module_non_scriptable.py 2022-09-27T15:58:57.8250274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:58:57.8262089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:58:57.8553206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:58:57.8756849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:58:58.5560274Z ok (2.562s) 2022-09-27T15:58:58.5560564Z 2022-09-27T15:58:58.5561225Z ---------------------------------------------------------------------- 2022-09-27T15:58:58.5561487Z Ran 1 test in 2.562s 2022-09-27T15:58:58.5561672Z 2022-09-27T15:58:58.5561732Z OK 2022-09-27T15:58:58.5561825Z 2022-09-27T15:58:58.5561918Z Generating XML reports... 2022-09-27T15:58:58.5596283Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155855.xml 2022-09-27T15:58:59.9290689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:58:59.9291096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:58:59.9300412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:58:59.9300837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:00.0923563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmk62yj4g 2022-09-27T15:59:00.0925282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmk62yj4g/_remote_module_non_scriptable.py 2022-09-27T15:59:00.3574022Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:00.3583302Z 2022-09-27T15:59:00.3583436Z Running tests... 2022-09-27T15:59:00.3584507Z ---------------------------------------------------------------------- 2022-09-27T15:59:00.6039742Z test_multiple_backward_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22578 2022-09-27T15:59:00.6066524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22579 2022-09-27T15:59:00.6094481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22580 2022-09-27T15:59:00.6123889Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22581 2022-09-27T15:59:01.7369681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:01.7370285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:01.7424127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:01.7424757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:01.7462058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:01.7462710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:01.7529452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:01.7529971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:01.7772925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:01.7773660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:01.7832926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:01.7833550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:01.7871131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:01.7871772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:01.7932774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:01.7933396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:01.9051018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmumubg0 2022-09-27T15:59:01.9052193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmumubg0/_remote_module_non_scriptable.py 2022-09-27T15:59:01.9116249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqryyrwjd 2022-09-27T15:59:01.9116950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqryyrwjd/_remote_module_non_scriptable.py 2022-09-27T15:59:01.9168629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp06dp9bk_ 2022-09-27T15:59:01.9170248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp06dp9bk_/_remote_module_non_scriptable.py 2022-09-27T15:59:01.9310782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppawbs11l 2022-09-27T15:59:01.9311584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppawbs11l/_remote_module_non_scriptable.py 2022-09-27T15:59:02.1633729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:59:02.1734407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:59:02.1810912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:59:02.1965900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:59:04.0205657Z ok (3.662s) 2022-09-27T15:59:04.0205929Z 2022-09-27T15:59:04.0206421Z ---------------------------------------------------------------------- 2022-09-27T15:59:04.0206656Z Ran 1 test in 3.662s 2022-09-27T15:59:04.0206770Z 2022-09-27T15:59:04.0206831Z OK 2022-09-27T15:59:04.0206921Z 2022-09-27T15:59:04.0207014Z Generating XML reports... 2022-09-27T15:59:04.0244351Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155900.xml 2022-09-27T15:59:05.3935679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:05.3936052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:05.3944777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:05.3945287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:05.5553877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzijpf2l 2022-09-27T15:59:05.5555069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzijpf2l/_remote_module_non_scriptable.py 2022-09-27T15:59:05.8241901Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:05.8251277Z 2022-09-27T15:59:05.8251637Z Running tests... 2022-09-27T15:59:05.8252039Z ---------------------------------------------------------------------- 2022-09-27T15:59:06.0725228Z test_nested_backward_accumulate_grads_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22833 2022-09-27T15:59:06.0755027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22834 2022-09-27T15:59:06.0800191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22835 2022-09-27T15:59:06.0847217Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22836 2022-09-27T15:59:07.1463572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:07.1464194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:07.1756459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:07.1757034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:07.1827953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:07.1828622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:07.1966873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:07.1967466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:07.2039484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:07.2040154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:07.2132910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:07.2133443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:07.2337286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:07.2337954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:07.2405774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:07.2406450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:07.3106467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqjhlovxx 2022-09-27T15:59:07.3107289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqjhlovxx/_remote_module_non_scriptable.py 2022-09-27T15:59:07.3394454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29yr_yx6 2022-09-27T15:59:07.3395440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29yr_yx6/_remote_module_non_scriptable.py 2022-09-27T15:59:07.3602346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93jfs5f1 2022-09-27T15:59:07.3603185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93jfs5f1/_remote_module_non_scriptable.py 2022-09-27T15:59:07.3671413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxo6u519 2022-09-27T15:59:07.3672326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxo6u519/_remote_module_non_scriptable.py 2022-09-27T15:59:07.5676508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:59:07.5946898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:59:07.6139876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:59:07.6223442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:59:08.2901715Z ok (2.465s) 2022-09-27T15:59:08.2901980Z 2022-09-27T15:59:08.2902468Z ---------------------------------------------------------------------- 2022-09-27T15:59:08.2902758Z Ran 1 test in 2.465s 2022-09-27T15:59:08.2903151Z 2022-09-27T15:59:08.2903389Z OK 2022-09-27T15:59:08.2903624Z 2022-09-27T15:59:08.2903803Z Generating XML reports... 2022-09-27T15:59:08.2938145Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155905.xml 2022-09-27T15:59:09.6591225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:09.6591921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:09.6601056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:09.6601660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:09.8197076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfkloj39 2022-09-27T15:59:09.8198249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfkloj39/_remote_module_non_scriptable.py 2022-09-27T15:59:10.0844562Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:10.0854823Z 2022-09-27T15:59:10.0855274Z Running tests... 2022-09-27T15:59:10.0855706Z ---------------------------------------------------------------------- 2022-09-27T15:59:10.3323425Z test_no_graph_with_tensors_not_require_grad_remote_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23088 2022-09-27T15:59:10.3352186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23089 2022-09-27T15:59:10.3380376Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23090 2022-09-27T15:59:10.3409475Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23091 2022-09-27T15:59:11.4658502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:11.4659172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:11.4777027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:11.4777668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:11.4810578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:11.4811209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:11.5038468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:11.5039130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:11.5054063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:11.5054765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:11.5178434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:11.5178976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:11.5213273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:11.5213812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:11.5430530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:11.5431205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:11.6329908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjvhygf2 2022-09-27T15:59:11.6330749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjvhygf2/_remote_module_non_scriptable.py 2022-09-27T15:59:11.6478834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4l752sy 2022-09-27T15:59:11.6479697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4l752sy/_remote_module_non_scriptable.py 2022-09-27T15:59:11.6492155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkp7ehsb7 2022-09-27T15:59:11.6493287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkp7ehsb7/_remote_module_non_scriptable.py 2022-09-27T15:59:11.6695042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_bonoxv 2022-09-27T15:59:11.6696454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_bonoxv/_remote_module_non_scriptable.py 2022-09-27T15:59:11.8962224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:59:11.9102838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:59:11.9149161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:59:11.9313715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:59:12.2030040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:59:12.2132717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:59:12.2133541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:59:12.2134339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:59:12.2135564Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:59:12.2136694Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:59:12.2137821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:59:12.2160938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:59:12.8473130Z ok (2.761s) 2022-09-27T15:59:12.8473351Z 2022-09-27T15:59:12.8473688Z ---------------------------------------------------------------------- 2022-09-27T15:59:12.8473941Z Ran 1 test in 2.762s 2022-09-27T15:59:12.8474110Z 2022-09-27T15:59:12.8474189Z OK 2022-09-27T15:59:12.8474281Z 2022-09-27T15:59:12.8474394Z Generating XML reports... 2022-09-27T15:59:12.8509627Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155910.xml 2022-09-27T15:59:14.2247750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:14.2248144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:14.2255737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:14.2256101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:14.3809862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jhd5g4l 2022-09-27T15:59:14.3810634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jhd5g4l/_remote_module_non_scriptable.py 2022-09-27T15:59:14.6434458Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:14.6444990Z 2022-09-27T15:59:14.6445387Z Running tests... 2022-09-27T15:59:14.6445947Z ---------------------------------------------------------------------- 2022-09-27T15:59:14.8881811Z test_no_graph_with_tensors_not_require_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23335 2022-09-27T15:59:14.8908585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23336 2022-09-27T15:59:14.8936962Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23337 2022-09-27T15:59:14.8966668Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23338 2022-09-27T15:59:15.9769483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:15.9770126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:15.9856268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:15.9856812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:15.9942716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:15.9943272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:16.0141762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:16.0142338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:16.0235361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:16.0235963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:16.0248835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:16.0249485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:16.0315206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:16.0315775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:16.0588097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:16.0588750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:16.1402815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa81jqc9t 2022-09-27T15:59:16.1404045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa81jqc9t/_remote_module_non_scriptable.py 2022-09-27T15:59:16.1497397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadfanrs_ 2022-09-27T15:59:16.1498377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadfanrs_/_remote_module_non_scriptable.py 2022-09-27T15:59:16.1568931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3q36wu6v 2022-09-27T15:59:16.1570251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3q36wu6v/_remote_module_non_scriptable.py 2022-09-27T15:59:16.1841361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgff__te 2022-09-27T15:59:16.1842376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgff__te/_remote_module_non_scriptable.py 2022-09-27T15:59:16.3996361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:59:16.4111859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:59:16.4158907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:59:16.4432389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:59:16.6908706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:59:16.7008335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:59:16.7111298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:59:16.7119547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:59:16.7121094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:59:16.7122553Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:59:16.7123550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:59:16.7124940Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:59:17.3026579Z ok (2.658s) 2022-09-27T15:59:17.3026826Z 2022-09-27T15:59:17.3027725Z ---------------------------------------------------------------------- 2022-09-27T15:59:17.3027974Z Ran 1 test in 2.658s 2022-09-27T15:59:17.3028154Z 2022-09-27T15:59:17.3028213Z OK 2022-09-27T15:59:17.3028308Z 2022-09-27T15:59:17.3028406Z Generating XML reports... 2022-09-27T15:59:17.3064656Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155914.xml 2022-09-27T15:59:18.6724750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:18.6732680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:18.6733193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:18.6733576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:18.8322748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tfvhdfr 2022-09-27T15:59:18.8323455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tfvhdfr/_remote_module_non_scriptable.py 2022-09-27T15:59:19.0953358Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:19.0963805Z 2022-09-27T15:59:19.0964078Z Running tests... 2022-09-27T15:59:19.0964747Z ---------------------------------------------------------------------- 2022-09-27T15:59:19.3466232Z test_remote_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23582 2022-09-27T15:59:19.3493814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23583 2022-09-27T15:59:19.3522041Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23584 2022-09-27T15:59:19.3553009Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23585 2022-09-27T15:59:20.5356469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:20.5357102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:20.5398498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:20.5399184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:20.5399995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:20.5400605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:20.5401396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:20.5402020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:20.5759171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:20.5759926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:20.5801270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:20.5802018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:20.5807635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:20.5808169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:20.5905710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:20.5906374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:20.7030289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pgk5r86 2022-09-27T15:59:20.7031367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pgk5r86/_remote_module_non_scriptable.py 2022-09-27T15:59:20.7057874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpq_b8xd 2022-09-27T15:59:20.7059028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpq_b8xd/_remote_module_non_scriptable.py 2022-09-27T15:59:20.7076665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfk963eh1 2022-09-27T15:59:20.7078327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfk963eh1/_remote_module_non_scriptable.py 2022-09-27T15:59:20.7206138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7o20lti 2022-09-27T15:59:20.7207471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7o20lti/_remote_module_non_scriptable.py 2022-09-27T15:59:20.9588998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:59:20.9600043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:59:20.9629319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:59:20.9796054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:59:21.8613451Z ok (2.765s) 2022-09-27T15:59:21.8613687Z 2022-09-27T15:59:21.8614099Z ---------------------------------------------------------------------- 2022-09-27T15:59:21.8614506Z Ran 1 test in 2.765s 2022-09-27T15:59:21.8614701Z 2022-09-27T15:59:21.8614801Z OK 2022-09-27T15:59:21.8614948Z 2022-09-27T15:59:21.8615088Z Generating XML reports... 2022-09-27T15:59:21.8651620Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155919.xml 2022-09-27T15:59:23.2437654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:23.2438052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:23.2446685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:23.2447056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:23.4035921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6f2ze79k 2022-09-27T15:59:23.4037321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6f2ze79k/_remote_module_non_scriptable.py 2022-09-27T15:59:23.6661407Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:23.6671011Z 2022-09-27T15:59:23.6671157Z Running tests... 2022-09-27T15:59:23.6671537Z ---------------------------------------------------------------------- 2022-09-27T15:59:23.9105265Z test_rpc_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23829 2022-09-27T15:59:23.9132018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23830 2022-09-27T15:59:23.9159772Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23831 2022-09-27T15:59:23.9188864Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23832 2022-09-27T15:59:25.1074305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:25.1075152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:25.1296044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:25.1297075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:25.1325257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:25.1326213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:25.1360934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:25.1361798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:25.1479279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:25.1479819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:25.1693828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:25.1694580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:25.1725653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:25.1726311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:25.1750431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:25.1751191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:25.2744149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1588ahnd 2022-09-27T15:59:25.2744889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1588ahnd/_remote_module_non_scriptable.py 2022-09-27T15:59:25.2960673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfl587lx6 2022-09-27T15:59:25.2961888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfl587lx6/_remote_module_non_scriptable.py 2022-09-27T15:59:25.3000216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4mansgt 2022-09-27T15:59:25.3002179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4mansgt/_remote_module_non_scriptable.py 2022-09-27T15:59:25.3013826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7k7iiilc 2022-09-27T15:59:25.3015432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7k7iiilc/_remote_module_non_scriptable.py 2022-09-27T15:59:25.5310141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:59:25.5534856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:59:25.5601056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:59:25.5614320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:59:26.3250408Z ok (2.657s) 2022-09-27T15:59:26.3250667Z 2022-09-27T15:59:26.3251161Z ---------------------------------------------------------------------- 2022-09-27T15:59:26.3251585Z Ran 1 test in 2.658s 2022-09-27T15:59:26.3251707Z 2022-09-27T15:59:26.3251784Z OK 2022-09-27T15:59:26.3251863Z 2022-09-27T15:59:26.3251961Z Generating XML reports... 2022-09-27T15:59:26.3286784Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155923.xml 2022-09-27T15:59:27.6929246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:27.6929673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:27.6939019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:27.6939377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:27.8566945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8akvu78 2022-09-27T15:59:27.8568283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8akvu78/_remote_module_non_scriptable.py 2022-09-27T15:59:28.1195187Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:28.1204750Z 2022-09-27T15:59:28.1204903Z Running tests... 2022-09-27T15:59:28.1205831Z ---------------------------------------------------------------------- 2022-09-27T15:59:28.3657497Z test_trainer_ps_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24076 2022-09-27T15:59:28.3683653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24077 2022-09-27T15:59:28.3711927Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24078 2022-09-27T15:59:28.3741648Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24079 2022-09-27T15:59:29.4862287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:29.4862937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:29.4871503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:29.4872112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:29.5235544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:29.5236308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:29.5245774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:29.5246387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:29.5268930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:29.5269592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:29.5281794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:29.5282416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:29.5629480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:29.5630072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:29.5659249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:29.5659867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:29.6503730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvon0jv0h 2022-09-27T15:59:29.6504520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvon0jv0h/_remote_module_non_scriptable.py 2022-09-27T15:59:29.6523343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7s8pvmy 2022-09-27T15:59:29.6525182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7s8pvmy/_remote_module_non_scriptable.py 2022-09-27T15:59:29.6903332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvo18cufr 2022-09-27T15:59:29.6904787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvo18cufr/_remote_module_non_scriptable.py 2022-09-27T15:59:29.6925069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5yajgnrb 2022-09-27T15:59:29.6926829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5yajgnrb/_remote_module_non_scriptable.py 2022-09-27T15:59:29.9036826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:59:29.9246350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:59:29.9461670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:59:29.9483037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:59:30.8804782Z ok (2.760s) 2022-09-27T15:59:30.8805053Z 2022-09-27T15:59:30.8805544Z ---------------------------------------------------------------------- 2022-09-27T15:59:30.8805783Z Ran 1 test in 2.760s 2022-09-27T15:59:30.8805901Z 2022-09-27T15:59:30.8805961Z OK 2022-09-27T15:59:30.8806053Z 2022-09-27T15:59:30.8806146Z Generating XML reports... 2022-09-27T15:59:30.8841238Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155928.xml 2022-09-27T15:59:32.2528050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:32.2528436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:32.2536644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:32.2536996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:32.4108687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp895ldoj3 2022-09-27T15:59:32.4109705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp895ldoj3/_remote_module_non_scriptable.py 2022-09-27T15:59:32.6722831Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:32.6732079Z 2022-09-27T15:59:32.6732217Z Running tests... 2022-09-27T15:59:32.6732657Z ---------------------------------------------------------------------- 2022-09-27T15:59:32.9188388Z test_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24343 2022-09-27T15:59:32.9214802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24344 2022-09-27T15:59:32.9243037Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24345 2022-09-27T15:59:32.9271678Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24346 2022-09-27T15:59:34.0154796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:34.0155482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:34.0444568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:34.0445213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:34.0525645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:34.0526298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:34.0554906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:34.0555557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:34.0585225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:34.0585870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:34.0828357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:34.0828957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:34.0939138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:34.0939871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:34.0963858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:34.0964511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:34.1801364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0kwd5czg 2022-09-27T15:59:34.1802177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0kwd5czg/_remote_module_non_scriptable.py 2022-09-27T15:59:34.2073296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppprlvlz9 2022-09-27T15:59:34.2074099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppprlvlz9/_remote_module_non_scriptable.py 2022-09-27T15:59:34.2197414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytgyckhs 2022-09-27T15:59:34.2198698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytgyckhs/_remote_module_non_scriptable.py 2022-09-27T15:59:34.2208017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp115cdvva 2022-09-27T15:59:34.2209998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp115cdvva/_remote_module_non_scriptable.py 2022-09-27T15:59:34.4358560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:59:34.4625819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:59:34.4737859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:59:34.4775990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:59:35.1327888Z ok (2.459s) 2022-09-27T15:59:35.1328140Z 2022-09-27T15:59:35.1328655Z ---------------------------------------------------------------------- 2022-09-27T15:59:35.1328927Z Ran 1 test in 2.459s 2022-09-27T15:59:35.1329045Z 2022-09-27T15:59:35.1329105Z OK 2022-09-27T15:59:35.1329197Z 2022-09-27T15:59:35.1329280Z Generating XML reports... 2022-09-27T15:59:35.1364699Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155932.xml 2022-09-27T15:59:36.4954404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:36.4954966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:36.4962907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:36.4963302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:36.6531114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1z75tw8 2022-09-27T15:59:36.6531560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1z75tw8/_remote_module_non_scriptable.py 2022-09-27T15:59:36.9169317Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:36.9178313Z 2022-09-27T15:59:36.9178526Z Running tests... 2022-09-27T15:59:36.9178884Z ---------------------------------------------------------------------- 2022-09-27T15:59:37.1636395Z test_builtin_remote_self_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24578 2022-09-27T15:59:37.1663456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24579 2022-09-27T15:59:37.1691151Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24580 2022-09-27T15:59:37.1719994Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24581 2022-09-27T15:59:38.3255830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:38.3256531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:38.3439797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:38.3440447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:38.3487793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:38.3634073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:38.3635226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:38.3635831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:38.3791695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:38.3792333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:38.3830034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:38.3830708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:38.3876443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:38.3877148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:38.4163611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:38.4164290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:38.4903929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmu62ig8t 2022-09-27T15:59:38.4904890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmu62ig8t/_remote_module_non_scriptable.py 2022-09-27T15:59:38.5092007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmxt2etv7 2022-09-27T15:59:38.5093045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmxt2etv7/_remote_module_non_scriptable.py 2022-09-27T15:59:38.5131454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0dqtojf9 2022-09-27T15:59:38.5133094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0dqtojf9/_remote_module_non_scriptable.py 2022-09-27T15:59:38.5420946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2i37spx 2022-09-27T15:59:38.5421912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2i37spx/_remote_module_non_scriptable.py 2022-09-27T15:59:38.7484788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:59:38.7663520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:59:38.7694117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:59:38.7991892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:59:39.4777435Z ok (2.559s) 2022-09-27T15:59:39.4777669Z 2022-09-27T15:59:39.4778137Z ---------------------------------------------------------------------- 2022-09-27T15:59:39.4778395Z Ran 1 test in 2.560s 2022-09-27T15:59:39.4778513Z 2022-09-27T15:59:39.4778574Z OK 2022-09-27T15:59:39.4778666Z 2022-09-27T15:59:39.4778747Z Generating XML reports... 2022-09-27T15:59:39.4813788Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155936.xml 2022-09-27T15:59:40.8598829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:40.8599218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:40.8606882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:40.8607231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:41.0175387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mwp6ost 2022-09-27T15:59:41.0176169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mwp6ost/_remote_module_non_scriptable.py 2022-09-27T15:59:41.2786106Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:41.2795924Z 2022-09-27T15:59:41.2796418Z Running tests... 2022-09-27T15:59:41.2796800Z ---------------------------------------------------------------------- 2022-09-27T15:59:41.5267005Z test_dynamic_and_static_init_rpc_together (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24813 2022-09-27T15:59:41.5295614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24814 2022-09-27T15:59:41.5323538Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24815 2022-09-27T15:59:41.5352969Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24816 2022-09-27T15:59:42.6882231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:42.6882905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:42.6890335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:42.6890980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:42.6925972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:42.6926624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:42.7066249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:42.7066796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:42.7268969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:42.7269527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:42.7279967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:42.7280467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:42.7316267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:42.7316771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:42.7446296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:42.7446896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:42.8535278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcfj2vhw 2022-09-27T15:59:42.8536001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcfj2vhw/_remote_module_non_scriptable.py 2022-09-27T15:59:42.8540075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdj143vd 2022-09-27T15:59:42.8542607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdj143vd/_remote_module_non_scriptable.py 2022-09-27T15:59:42.8571571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30uflamb 2022-09-27T15:59:42.8573051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30uflamb/_remote_module_non_scriptable.py 2022-09-27T15:59:42.8712914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7oftfwfo 2022-09-27T15:59:42.8714715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7oftfwfo/_remote_module_non_scriptable.py 2022-09-27T15:59:43.1073867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:59:43.1079838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:59:43.1123208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:59:43.1260288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:59:43.1414740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T15:59:43.1617023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T15:59:43.1617941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T15:59:43.1618499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T15:59:43.1619414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:59:43.1620209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:59:43.1621028Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:59:43.1621845Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T15:59:43.7082341Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:59:43.7841188Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T15:59:43.8415147Z ok (2.562s) 2022-09-27T15:59:43.8415414Z 2022-09-27T15:59:43.8415804Z ---------------------------------------------------------------------- 2022-09-27T15:59:43.8416064Z Ran 1 test in 2.562s 2022-09-27T15:59:43.8416190Z 2022-09-27T15:59:43.8416286Z OK 2022-09-27T15:59:43.8416376Z 2022-09-27T15:59:43.8417352Z Generating XML reports... 2022-09-27T15:59:43.8453157Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155941.xml 2022-09-27T15:59:45.2157898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:45.2158283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:45.2165686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:45.2166044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:45.3730100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9kz939id 2022-09-27T15:59:45.3732162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9kz939id/_remote_module_non_scriptable.py 2022-09-27T15:59:45.6343328Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:45.6352376Z 2022-09-27T15:59:45.6352474Z Running tests... 2022-09-27T15:59:45.6353811Z ---------------------------------------------------------------------- 2022-09-27T15:59:45.8707606Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84116 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.235s) 2022-09-27T15:59:45.8708154Z 2022-09-27T15:59:45.8708595Z ---------------------------------------------------------------------- 2022-09-27T15:59:45.8708857Z Ran 1 test in 0.235s 2022-09-27T15:59:45.8708956Z 2022-09-27T15:59:45.8709097Z OK (skipped=1) 2022-09-27T15:59:45.8709203Z 2022-09-27T15:59:45.8709287Z Generating XML reports... 2022-09-27T15:59:45.8733336Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155945.xml 2022-09-27T15:59:47.1839811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:47.1840188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:47.1848188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:47.1848707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:47.3436040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx45u3_ns 2022-09-27T15:59:47.3437768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx45u3_ns/_remote_module_non_scriptable.py 2022-09-27T15:59:47.6068794Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:47.6078676Z 2022-09-27T15:59:47.6078786Z Running tests... 2022-09-27T15:59:47.6079380Z ---------------------------------------------------------------------- 2022-09-27T15:59:47.8529712Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25033 2022-09-27T15:59:47.8557425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25034 2022-09-27T15:59:47.8585329Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25035 2022-09-27T15:59:47.8615466Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25036 2022-09-27T15:59:48.9508805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:48.9509424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:48.9582000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:48.9582696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:48.9583495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:48.9584122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:48.9804196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:48.9804845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:48.9886878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:48.9887497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:48.9962269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:48.9962918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:48.9963728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:48.9964340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:49.0160313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:49.0160963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:49.1155165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgfoaizgn 2022-09-27T15:59:49.1155870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgfoaizgn/_remote_module_non_scriptable.py 2022-09-27T15:59:49.1220520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_qbjkcr 2022-09-27T15:59:49.1221224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfb29zy4l 2022-09-27T15:59:49.1221946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_qbjkcr/_remote_module_non_scriptable.py 2022-09-27T15:59:49.1223117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfb29zy4l/_remote_module_non_scriptable.py 2022-09-27T15:59:49.1406388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnrjjlak 2022-09-27T15:59:49.1407127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnrjjlak/_remote_module_non_scriptable.py 2022-09-27T15:59:49.3727415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:59:49.3763321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:59:49.3766956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:59:49.3947764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:59:49.6665115Z skip: Need at least 2 CUDA devices (2.058s) 2022-09-27T15:59:49.6665380Z 2022-09-27T15:59:49.6665899Z ---------------------------------------------------------------------- 2022-09-27T15:59:49.6666211Z Ran 1 test in 2.059s 2022-09-27T15:59:49.6666324Z 2022-09-27T15:59:49.6666396Z OK (skipped=1) 2022-09-27T15:59:49.6666504Z 2022-09-27T15:59:49.6666575Z Generating XML reports... 2022-09-27T15:59:49.6701024Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155947.xml 2022-09-27T15:59:51.0467401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:51.0467845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:51.0474422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:51.0475008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:51.2078332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8a35cvl 2022-09-27T15:59:51.2079200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8a35cvl/_remote_module_non_scriptable.py 2022-09-27T15:59:51.4736007Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:51.4745624Z 2022-09-27T15:59:51.4745765Z Running tests... 2022-09-27T15:59:51.4746189Z ---------------------------------------------------------------------- 2022-09-27T15:59:51.7240990Z test_dynamic_rpc_init_rpc (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25088 2022-09-27T15:59:51.7270239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25089 2022-09-27T15:59:51.7299263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25090 2022-09-27T15:59:51.7329305Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25091 2022-09-27T15:59:52.8266244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:52.8266907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:52.8286092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:52.8286669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:52.8553501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:52.8554110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:52.8595698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:52.8596364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:52.8639618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:52.8640303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:52.8657565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:52.8658059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:52.8927449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:52.8928207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:52.8967716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:52.8968334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:52.9899479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp594h59bk 2022-09-27T15:59:52.9900283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp594h59bk/_remote_module_non_scriptable.py 2022-09-27T15:59:52.9902738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgx8oa8b 2022-09-27T15:59:52.9905147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgx8oa8b/_remote_module_non_scriptable.py 2022-09-27T15:59:53.0178757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xo3ye07 2022-09-27T15:59:53.0179634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xo3ye07/_remote_module_non_scriptable.py 2022-09-27T15:59:53.0225031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph284gwlc 2022-09-27T15:59:53.0226328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph284gwlc/_remote_module_non_scriptable.py 2022-09-27T15:59:53.2457454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:59:53.2460501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:59:53.2761026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:59:53.2822640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:59:53.9383991Z ok (2.463s) 2022-09-27T15:59:53.9384342Z 2022-09-27T15:59:53.9384818Z ---------------------------------------------------------------------- 2022-09-27T15:59:53.9385142Z Ran 1 test in 2.464s 2022-09-27T15:59:53.9385278Z 2022-09-27T15:59:53.9385339Z OK 2022-09-27T15:59:53.9385432Z 2022-09-27T15:59:53.9385527Z Generating XML reports... 2022-09-27T15:59:53.9422344Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155951.xml 2022-09-27T15:59:55.3095235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:55.3095642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:55.3103213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:55.3103575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:55.4704686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp102ww1ol 2022-09-27T15:59:55.4705485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp102ww1ol/_remote_module_non_scriptable.py 2022-09-27T15:59:55.7336765Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:55.7346041Z 2022-09-27T15:59:55.7346181Z Running tests... 2022-09-27T15:59:55.7346769Z ---------------------------------------------------------------------- 2022-09-27T15:59:55.9775345Z test_dynamic_rpc_init_rpc_without_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25299 2022-09-27T15:59:55.9802535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25300 2022-09-27T15:59:55.9830525Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25301 2022-09-27T15:59:55.9859816Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25302 2022-09-27T15:59:57.1422630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:57.1423265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:57.1801456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:57.1802116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:57.2074747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:57.2075390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:57.2151617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:57.2152157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:57.2463453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:57.2464098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:57.2530865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:57.2531602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:57.2570518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:57.2571240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:57.2956500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:57.2957125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:57.3092767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pzrvlzx 2022-09-27T15:59:57.3093841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pzrvlzx/_remote_module_non_scriptable.py 2022-09-27T15:59:57.3751899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpau8kpxgp 2022-09-27T15:59:57.3752702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpau8kpxgp/_remote_module_non_scriptable.py 2022-09-27T15:59:57.3818195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd24pu9mw 2022-09-27T15:59:57.3819108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd24pu9mw/_remote_module_non_scriptable.py 2022-09-27T15:59:57.4229320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2_fvpnr 2022-09-27T15:59:57.4230516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2_fvpnr/_remote_module_non_scriptable.py 2022-09-27T15:59:57.5695124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T15:59:57.6370221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T15:59:57.6425515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T15:59:57.6808775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T15:59:57.9913620Z ok (2.256s) 2022-09-27T15:59:57.9913848Z 2022-09-27T15:59:57.9914321Z ---------------------------------------------------------------------- 2022-09-27T15:59:57.9914893Z Ran 1 test in 2.257s 2022-09-27T15:59:57.9915010Z 2022-09-27T15:59:57.9915071Z OK 2022-09-27T15:59:57.9915167Z 2022-09-27T15:59:57.9915250Z Generating XML reports... 2022-09-27T15:59:57.9949580Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155955.xml 2022-09-27T15:59:59.3610165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T15:59:59.3610546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T15:59:59.3618268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T15:59:59.3618705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T15:59:59.5167718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajr0t1v9 2022-09-27T15:59:59.5168807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajr0t1v9/_remote_module_non_scriptable.py 2022-09-27T15:59:59.7798132Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T15:59:59.7808397Z 2022-09-27T15:59:59.7808807Z Running tests... 2022-09-27T15:59:59.7809200Z ---------------------------------------------------------------------- 2022-09-27T16:00:00.0254519Z test_dynamic_rpc_new_rank_can_communicated_with_existing_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25358 2022-09-27T16:00:00.0280298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25359 2022-09-27T16:00:00.0307971Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25360 2022-09-27T16:00:00.0337517Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25361 2022-09-27T16:00:01.1421214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:01.1421874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:01.1499380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:01.1500014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:01.1759733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:01.1760276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:01.1783828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:01.1784552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:01.1813894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:01.1814431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:01.1898237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:01.1898904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:01.2156668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:01.2157283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:01.2173672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:01.2174346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:01.3077206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpet3spohk 2022-09-27T16:00:01.3078389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpet3spohk/_remote_module_non_scriptable.py 2022-09-27T16:00:01.3165080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqimioo7t 2022-09-27T16:00:01.3166495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqimioo7t/_remote_module_non_scriptable.py 2022-09-27T16:00:01.3432650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinh00i92 2022-09-27T16:00:01.3433358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinh00i92/_remote_module_non_scriptable.py 2022-09-27T16:00:01.3436321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgiyfw8q 2022-09-27T16:00:01.3438332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgiyfw8q/_remote_module_non_scriptable.py 2022-09-27T16:00:01.5703551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:01.5781176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:01.6076474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:01.6083454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:01.6267560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:00:01.6368339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:00:01.6449407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:00:01.6449825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:00:01.6450520Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:00:01.6451103Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:00:01.6470882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:00:01.6471770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:00:02.2048211Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T16:00:02.2049463Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T16:00:02.2050671Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T16:00:02.2212418Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T16:00:02.2213656Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T16:00:02.2279119Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T16:00:02.6402435Z ok (2.859s) 2022-09-27T16:00:02.6402634Z 2022-09-27T16:00:02.6403248Z ---------------------------------------------------------------------- 2022-09-27T16:00:02.6403489Z Ran 1 test in 2.859s 2022-09-27T16:00:02.6403604Z 2022-09-27T16:00:02.6403664Z OK 2022-09-27T16:00:02.6403757Z 2022-09-27T16:00:02.6403849Z Generating XML reports... 2022-09-27T16:00:02.6438760Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155959.xml 2022-09-27T16:00:04.0303077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:04.0303706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:04.0312986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:04.0313640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:04.1907674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptcmp_w_6 2022-09-27T16:00:04.1908354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptcmp_w_6/_remote_module_non_scriptable.py 2022-09-27T16:00:04.4564321Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:00:04.4574621Z 2022-09-27T16:00:04.4574957Z Running tests... 2022-09-27T16:00:04.4575566Z ---------------------------------------------------------------------- 2022-09-27T16:00:04.7043069Z test_infer_backend_from_options (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25581 2022-09-27T16:00:04.7071295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25582 2022-09-27T16:00:04.7099313Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25583 2022-09-27T16:00:04.7129152Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25584 2022-09-27T16:00:05.8585946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:05.8586645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:05.8713532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:05.8714043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:05.8745947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:05.8746483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:05.8769934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:05.8770595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:05.8986439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:05.8987002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:05.9110393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:05.9111199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:05.9142813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:05.9143525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:05.9170135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:05.9170692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:06.0304041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurkjwahq 2022-09-27T16:00:06.0304845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurkjwahq/_remote_module_non_scriptable.py 2022-09-27T16:00:06.0415572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6c07n3p 2022-09-27T16:00:06.0416524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6c07n3p/_remote_module_non_scriptable.py 2022-09-27T16:00:06.0444331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4zq_2g4b 2022-09-27T16:00:06.0446149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4zq_2g4b/_remote_module_non_scriptable.py 2022-09-27T16:00:06.0465235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkd6hrb0 2022-09-27T16:00:06.0466602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkd6hrb0/_remote_module_non_scriptable.py 2022-09-27T16:00:06.3026453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:06.3078325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:06.3090686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:06.3128485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:07.0432162Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T16:00:07.0433079Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T16:00:07.0434022Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T16:00:07.1190245Z ok (2.661s) 2022-09-27T16:00:07.1190518Z 2022-09-27T16:00:07.1191038Z ---------------------------------------------------------------------- 2022-09-27T16:00:07.1191422Z Ran 1 test in 2.661s 2022-09-27T16:00:07.1191621Z 2022-09-27T16:00:07.1191715Z OK 2022-09-27T16:00:07.1191983Z 2022-09-27T16:00:07.1192129Z Generating XML reports... 2022-09-27T16:00:07.1228141Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160004.xml 2022-09-27T16:00:08.4927530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:08.4928162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:08.4936144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:08.4936535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:08.6509288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5ply14n 2022-09-27T16:00:08.6510126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5ply14n/_remote_module_non_scriptable.py 2022-09-27T16:00:08.9130578Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:00:08.9139628Z 2022-09-27T16:00:08.9139764Z Running tests... 2022-09-27T16:00:08.9140151Z ---------------------------------------------------------------------- 2022-09-27T16:00:09.1634726Z test_mismatched_type_for_options (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25804 2022-09-27T16:00:09.1661386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25805 2022-09-27T16:00:09.1697592Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25806 2022-09-27T16:00:09.1734032Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25807 2022-09-27T16:00:10.2653164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:10.2653784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:10.2860110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:10.2860770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:10.3029126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:10.3029655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:10.3065451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:10.3065958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:10.3174258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:10.3174975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:10.3239381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:10.3239997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:10.3445423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:10.3446097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:10.3552195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:10.3552865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:10.4300562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppx3a7i9z 2022-09-27T16:00:10.4301827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppx3a7i9z/_remote_module_non_scriptable.py 2022-09-27T16:00:10.4498093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi3m6q6n0 2022-09-27T16:00:10.4499009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi3m6q6n0/_remote_module_non_scriptable.py 2022-09-27T16:00:10.4705436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6tqvm2xw 2022-09-27T16:00:10.4706337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6tqvm2xw/_remote_module_non_scriptable.py 2022-09-27T16:00:10.4806821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxx8y96nn 2022-09-27T16:00:10.4807887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxx8y96nn/_remote_module_non_scriptable.py 2022-09-27T16:00:10.6891087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:10.7052906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:10.7265429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:10.7389136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:11.0784091Z ok (2.164s) 2022-09-27T16:00:11.0784332Z 2022-09-27T16:00:11.0784835Z ---------------------------------------------------------------------- 2022-09-27T16:00:11.0785155Z Ran 1 test in 2.164s 2022-09-27T16:00:11.0785272Z 2022-09-27T16:00:11.0785334Z OK 2022-09-27T16:00:11.0785427Z 2022-09-27T16:00:11.0785525Z Generating XML reports... 2022-09-27T16:00:11.0822537Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160008.xml 2022-09-27T16:00:12.4704137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:12.4704780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:12.4712693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:12.4713290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:12.6330539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo33yprkb 2022-09-27T16:00:12.6331362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo33yprkb/_remote_module_non_scriptable.py 2022-09-27T16:00:12.8973741Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:00:12.8983345Z 2022-09-27T16:00:12.8983478Z Running tests... 2022-09-27T16:00:12.8984113Z ---------------------------------------------------------------------- 2022-09-27T16:00:13.1426013Z test_multi_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25859 2022-09-27T16:00:13.1455857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25860 2022-09-27T16:00:13.1483911Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25861 2022-09-27T16:00:13.1515114Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25862 2022-09-27T16:00:14.2295164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:14.2295763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:14.2569514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:14.2570157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:14.2686799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:14.2687405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:14.2747235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:14.2747876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:14.2754007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:14.2754800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:14.2955544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:14.2956113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:14.3129782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:14.3130524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:14.3141065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:14.3141636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:14.3984008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprx6cuxu2 2022-09-27T16:00:14.3984743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprx6cuxu2/_remote_module_non_scriptable.py 2022-09-27T16:00:14.4403539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1u6ywqrc 2022-09-27T16:00:14.4404343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1u6ywqrc/_remote_module_non_scriptable.py 2022-09-27T16:00:14.4413671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdd5colln 2022-09-27T16:00:14.4415151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdd5colln/_remote_module_non_scriptable.py 2022-09-27T16:00:14.4438983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphdtzqupf 2022-09-27T16:00:14.4440486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphdtzqupf/_remote_module_non_scriptable.py 2022-09-27T16:00:14.6542828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:14.6953847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:14.6961587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:14.7013312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:15.4573916Z ok (2.559s) 2022-09-27T16:00:15.4574167Z 2022-09-27T16:00:15.4574570Z ---------------------------------------------------------------------- 2022-09-27T16:00:15.4574808Z Ran 1 test in 2.559s 2022-09-27T16:00:15.4574943Z 2022-09-27T16:00:15.4575036Z OK 2022-09-27T16:00:15.4575134Z 2022-09-27T16:00:15.4575230Z Generating XML reports... 2022-09-27T16:00:15.4610483Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160012.xml 2022-09-27T16:00:16.8467789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:16.8468476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:16.8477447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:16.8478085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:17.0039317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ofscuqr 2022-09-27T16:00:17.0040325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ofscuqr/_remote_module_non_scriptable.py 2022-09-27T16:00:17.2669937Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:00:17.2680562Z 2022-09-27T16:00:17.2680996Z Running tests... 2022-09-27T16:00:17.2681402Z ---------------------------------------------------------------------- 2022-09-27T16:00:17.5131159Z test_multi_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26094 2022-09-27T16:00:17.5158637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26095 2022-09-27T16:00:17.5186424Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26096 2022-09-27T16:00:17.5215880Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26097 2022-09-27T16:00:18.6278757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:18.6279450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:18.6526031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:18.6526557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:18.6654157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:18.6654738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:18.6864767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:18.6865403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:18.6901000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:18.6901683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:18.7030523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:18.7031119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:18.7233833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:18.7234869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:18.7390108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:18.7390726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:18.7922375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1potjvt2 2022-09-27T16:00:18.7923453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1potjvt2/_remote_module_non_scriptable.py 2022-09-27T16:00:18.8180237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5zqhrvk 2022-09-27T16:00:18.8181054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5zqhrvk/_remote_module_non_scriptable.py 2022-09-27T16:00:18.8500145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2k5w4ce 2022-09-27T16:00:18.8500970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2k5w4ce/_remote_module_non_scriptable.py 2022-09-27T16:00:18.8651669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqh_c1t86 2022-09-27T16:00:18.8652425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqh_c1t86/_remote_module_non_scriptable.py 2022-09-27T16:00:19.0491456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:19.0776161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:19.1073633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:19.1228846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:20.0276745Z ok (2.759s) 2022-09-27T16:00:20.0276950Z 2022-09-27T16:00:20.0277310Z ---------------------------------------------------------------------- 2022-09-27T16:00:20.0277590Z Ran 1 test in 2.760s 2022-09-27T16:00:20.0277722Z 2022-09-27T16:00:20.0277816Z OK 2022-09-27T16:00:20.0277926Z 2022-09-27T16:00:20.0278021Z Generating XML reports... 2022-09-27T16:00:20.0313417Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160017.xml 2022-09-27T16:00:21.3978823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:21.3979401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:21.4003609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:21.4004221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:21.5584617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2tm6wls 2022-09-27T16:00:21.5585763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2tm6wls/_remote_module_non_scriptable.py 2022-09-27T16:00:21.8200553Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:00:21.8210231Z 2022-09-27T16:00:21.8210336Z Running tests... 2022-09-27T16:00:21.8211237Z ---------------------------------------------------------------------- 2022-09-27T16:00:22.0676229Z test_multi_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26329 2022-09-27T16:00:22.0703906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26330 2022-09-27T16:00:22.0732298Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26331 2022-09-27T16:00:22.0761973Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26332 2022-09-27T16:00:23.1228017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:23.1228706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:23.1587267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:23.1587839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:23.1794295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:23.1795122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:23.1927445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:23.1928089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:23.2057595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:23.2058189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:23.2175559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:23.2176105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:23.2313063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:23.2313701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:23.2439016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:23.2439687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:23.2863716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1i9gbli7 2022-09-27T16:00:23.2864708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1i9gbli7/_remote_module_non_scriptable.py 2022-09-27T16:00:23.3459894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9qnvpmr 2022-09-27T16:00:23.3460690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9qnvpmr/_remote_module_non_scriptable.py 2022-09-27T16:00:23.3597637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdds8x6l2 2022-09-27T16:00:23.3598833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdds8x6l2/_remote_module_non_scriptable.py 2022-09-27T16:00:23.3714343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp309ygp5o 2022-09-27T16:00:23.3715795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp309ygp5o/_remote_module_non_scriptable.py 2022-09-27T16:00:23.5451097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:23.6083703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:23.6223546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:23.6313302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:24.6825201Z ok (2.861s) 2022-09-27T16:00:24.6825381Z 2022-09-27T16:00:24.6825861Z ---------------------------------------------------------------------- 2022-09-27T16:00:24.6826282Z Ran 1 test in 2.861s 2022-09-27T16:00:24.6826497Z 2022-09-27T16:00:24.6826578Z OK 2022-09-27T16:00:24.6826935Z 2022-09-27T16:00:24.6827081Z Generating XML reports... 2022-09-27T16:00:24.6861968Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160021.xml 2022-09-27T16:00:26.0635131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:26.0635759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:26.0643706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:26.0644353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:26.2222059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu14fysss 2022-09-27T16:00:26.2223159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu14fysss/_remote_module_non_scriptable.py 2022-09-27T16:00:26.4849397Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:00:26.4859040Z 2022-09-27T16:00:26.4859155Z Running tests... 2022-09-27T16:00:26.4859744Z ---------------------------------------------------------------------- 2022-09-27T16:00:26.7371563Z test_my_parameter_server_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26564 2022-09-27T16:00:26.7398921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26565 2022-09-27T16:00:26.7429707Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26566 2022-09-27T16:00:26.7458039Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26567 2022-09-27T16:00:27.8850728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:27.8851386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:27.8989306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:27.8990044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:27.9000273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:27.9000923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:27.9016215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:27.9016883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:27.9254327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:27.9254928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:27.9392554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:27.9393213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:27.9403301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:27.9403942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:27.9414781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:27.9415407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:28.0518773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqv1qmbit 2022-09-27T16:00:28.0519577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqv1qmbit/_remote_module_non_scriptable.py 2022-09-27T16:00:28.0673517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ibg7vfx 2022-09-27T16:00:28.0674253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzza62aeh 2022-09-27T16:00:28.0675128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ibg7vfx/_remote_module_non_scriptable.py 2022-09-27T16:00:28.0675758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzza62aeh/_remote_module_non_scriptable.py 2022-09-27T16:00:28.0678437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprk4zvwss 2022-09-27T16:00:28.0680591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprk4zvwss/_remote_module_non_scriptable.py 2022-09-27T16:00:28.3078429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:28.3242872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:28.3256367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:28.3284366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:29.1518129Z ok (2.665s) 2022-09-27T16:00:29.1518417Z 2022-09-27T16:00:29.1519069Z ---------------------------------------------------------------------- 2022-09-27T16:00:29.1519450Z Ran 1 test in 2.666s 2022-09-27T16:00:29.1519567Z 2022-09-27T16:00:29.1519632Z OK 2022-09-27T16:00:29.1519724Z 2022-09-27T16:00:29.1519816Z Generating XML reports... 2022-09-27T16:00:29.1555048Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160026.xml 2022-09-27T16:00:30.5320910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:30.5321296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:30.5328787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:30.5329212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:30.6919630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9fal2fy 2022-09-27T16:00:30.6920459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9fal2fy/_remote_module_non_scriptable.py 2022-09-27T16:00:30.9544236Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:00:30.9553957Z 2022-09-27T16:00:30.9554099Z Running tests... 2022-09-27T16:00:30.9554641Z ---------------------------------------------------------------------- 2022-09-27T16:00:31.1987364Z test_nested_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26823 2022-09-27T16:00:31.2015032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26824 2022-09-27T16:00:31.2043935Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26825 2022-09-27T16:00:31.2073436Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26826 2022-09-27T16:00:32.3202920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:32.3203554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:32.3209380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:32.3210010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:32.3390503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:32.3391226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:32.3585218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:32.3585804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:32.3597335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:32.3597953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:32.3643621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:32.3644268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:32.3774446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:32.3774954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:32.4014921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:32.4015549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:32.4853921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_q5cnmh 2022-09-27T16:00:32.4854871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_q5cnmh/_remote_module_non_scriptable.py 2022-09-27T16:00:32.4873175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvabqf8w 2022-09-27T16:00:32.4875866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvabqf8w/_remote_module_non_scriptable.py 2022-09-27T16:00:32.5039477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7i2ot5z 2022-09-27T16:00:32.5040348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7i2ot5z/_remote_module_non_scriptable.py 2022-09-27T16:00:32.5280570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprz1o8sq3 2022-09-27T16:00:32.5281344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprz1o8sq3/_remote_module_non_scriptable.py 2022-09-27T16:00:32.7417973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:32.7492622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:32.7620978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:32.7855554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:33.5131693Z ok (2.557s) 2022-09-27T16:00:33.5131968Z 2022-09-27T16:00:33.5132366Z ---------------------------------------------------------------------- 2022-09-27T16:00:33.5132603Z Ran 1 test in 2.558s 2022-09-27T16:00:33.5132726Z 2022-09-27T16:00:33.5132787Z OK 2022-09-27T16:00:33.5132879Z 2022-09-27T16:00:33.5132973Z Generating XML reports... 2022-09-27T16:00:33.5168697Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160030.xml 2022-09-27T16:00:34.8938626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:34.8939016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:34.8946920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:34.8947301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:35.0517737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5043w1r8 2022-09-27T16:00:35.0518725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5043w1r8/_remote_module_non_scriptable.py 2022-09-27T16:00:35.3137999Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:00:35.3147577Z 2022-09-27T16:00:35.3147669Z Running tests... 2022-09-27T16:00:35.3149042Z ---------------------------------------------------------------------- 2022-09-27T16:00:35.5626916Z test_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27070 2022-09-27T16:00:35.5654259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27071 2022-09-27T16:00:35.5681858Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27072 2022-09-27T16:00:35.5712531Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27073 2022-09-27T16:00:36.7136133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:36.7136766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:36.7495927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:36.7496511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:36.7816519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:36.7817217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:36.8109916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:36.8110501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:36.8174973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:36.8175638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:36.8474967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:36.8475596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:36.8623096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:36.8623808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:36.8765788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewohdovk 2022-09-27T16:00:36.8766976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewohdovk/_remote_module_non_scriptable.py 2022-09-27T16:00:36.9000396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:36.9001003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:36.9448231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28wiruk2 2022-09-27T16:00:36.9448895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28wiruk2/_remote_module_non_scriptable.py 2022-09-27T16:00:36.9750554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp393kxsa_ 2022-09-27T16:00:36.9751433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp393kxsa_/_remote_module_non_scriptable.py 2022-09-27T16:00:37.0265447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7tw7aojf 2022-09-27T16:00:37.0266123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7tw7aojf/_remote_module_non_scriptable.py 2022-09-27T16:00:37.1397111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:37.2044944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:37.2416392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:37.3080709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:38.0803827Z ok (2.765s) 2022-09-27T16:00:38.0804395Z 2022-09-27T16:00:38.0804825Z ---------------------------------------------------------------------- 2022-09-27T16:00:38.0805158Z Ran 1 test in 2.766s 2022-09-27T16:00:38.0805274Z 2022-09-27T16:00:38.0805337Z OK 2022-09-27T16:00:38.0805428Z 2022-09-27T16:00:38.0805508Z Generating XML reports... 2022-09-27T16:00:38.0841028Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160035.xml 2022-09-27T16:00:39.4691867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:39.4692252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:39.4700274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:39.4700629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:39.6284534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfb7htp09 2022-09-27T16:00:39.6285743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfb7htp09/_remote_module_non_scriptable.py 2022-09-27T16:00:39.8900976Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:00:39.8910692Z 2022-09-27T16:00:39.8910877Z Running tests... 2022-09-27T16:00:39.8911392Z ---------------------------------------------------------------------- 2022-09-27T16:00:40.1322279Z test_nested_rref_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27317 2022-09-27T16:00:40.1349682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27318 2022-09-27T16:00:40.1377325Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27319 2022-09-27T16:00:40.1406276Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27320 2022-09-27T16:00:41.2403511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:41.2404180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:41.2616320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:41.2616951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:41.2666780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:41.2667452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:41.2758179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:41.2758650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:41.2782220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:41.2782736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:41.3011995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:41.3012740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:41.3063445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:41.3064099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:41.3241939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:41.3242621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:41.4078586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgp5f_z3j 2022-09-27T16:00:41.4079389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgp5f_z3j/_remote_module_non_scriptable.py 2022-09-27T16:00:41.4290254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3c7diq2 2022-09-27T16:00:41.4291061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3c7diq2/_remote_module_non_scriptable.py 2022-09-27T16:00:41.4333025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehyfre6m 2022-09-27T16:00:41.4334399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehyfre6m/_remote_module_non_scriptable.py 2022-09-27T16:00:41.4511073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0gc572e 2022-09-27T16:00:41.4511899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0gc572e/_remote_module_non_scriptable.py 2022-09-27T16:00:41.6658592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:41.6871409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:41.6940101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:41.7127216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:42.4486436Z ok (2.557s) 2022-09-27T16:00:42.4486676Z 2022-09-27T16:00:42.4487185Z ---------------------------------------------------------------------- 2022-09-27T16:00:42.4487529Z Ran 1 test in 2.557s 2022-09-27T16:00:42.4487646Z 2022-09-27T16:00:42.4487706Z OK 2022-09-27T16:00:42.4487800Z 2022-09-27T16:00:42.4487892Z Generating XML reports... 2022-09-27T16:00:42.4527711Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160039.xml 2022-09-27T16:00:43.8183045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:43.8183432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:43.8191778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:43.8192147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:43.9776421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_rn024g 2022-09-27T16:00:43.9777415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_rn024g/_remote_module_non_scriptable.py 2022-09-27T16:00:44.2393111Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:00:44.2403444Z 2022-09-27T16:00:44.2403752Z Running tests... 2022-09-27T16:00:44.2404368Z ---------------------------------------------------------------------- 2022-09-27T16:00:44.4879273Z test_nested_rref_stress_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27564 2022-09-27T16:00:44.4906830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27565 2022-09-27T16:00:44.4935056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27566 2022-09-27T16:00:44.4966423Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27567 2022-09-27T16:00:45.6046152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:45.6046784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:45.6077211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:45.6077864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:45.6091188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:45.6092155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:45.6145768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:45.6146690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:45.6418512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:45.6419377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:45.6464939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:45.6465636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:45.6476275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:45.6476849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:45.6530128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:45.6530766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:45.7674462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8cpq2yi 2022-09-27T16:00:45.7675708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8cpq2yi/_remote_module_non_scriptable.py 2022-09-27T16:00:45.7719282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_lanlnw 2022-09-27T16:00:45.7720631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_lanlnw/_remote_module_non_scriptable.py 2022-09-27T16:00:45.7728529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_ldxrda 2022-09-27T16:00:45.7730821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_ldxrda/_remote_module_non_scriptable.py 2022-09-27T16:00:45.7789975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi72ln_cv 2022-09-27T16:00:45.7791930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi72ln_cv/_remote_module_non_scriptable.py 2022-09-27T16:00:46.0239887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:46.0274379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:46.0301023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:46.0391550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:46.9027117Z ok (2.662s) 2022-09-27T16:00:46.9027364Z 2022-09-27T16:00:46.9027905Z ---------------------------------------------------------------------- 2022-09-27T16:00:46.9028263Z Ran 1 test in 2.662s 2022-09-27T16:00:46.9028381Z 2022-09-27T16:00:46.9028447Z OK 2022-09-27T16:00:46.9028540Z 2022-09-27T16:00:46.9028636Z Generating XML reports... 2022-09-27T16:00:46.9064218Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160044.xml 2022-09-27T16:00:48.2976698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:48.2977359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:48.2985990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:48.2986631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:48.4560589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplwy0ny8s 2022-09-27T16:00:48.4561826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplwy0ny8s/_remote_module_non_scriptable.py 2022-09-27T16:00:48.7167601Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:00:48.7177711Z 2022-09-27T16:00:48.7178127Z Running tests... 2022-09-27T16:00:48.7178508Z ---------------------------------------------------------------------- 2022-09-27T16:00:48.9620501Z test_op_with_invalid_args (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27991 2022-09-27T16:00:48.9648329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27992 2022-09-27T16:00:48.9675928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27993 2022-09-27T16:00:48.9705231Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27994 2022-09-27T16:00:50.0721381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:50.0722011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:50.0759511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:50.0774786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:50.0795827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:50.0796477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:50.1099308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:50.1099937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:50.1136544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:50.1137180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:50.1158214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:50.1158827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:50.1168078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:50.1168708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:50.1505665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:50.1506275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:50.2371073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqujbsspg 2022-09-27T16:00:50.2371899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqujbsspg/_remote_module_non_scriptable.py 2022-09-27T16:00:50.2407771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52nwvuiy 2022-09-27T16:00:50.2408891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52nwvuiy/_remote_module_non_scriptable.py 2022-09-27T16:00:50.2427991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4ybyosp 2022-09-27T16:00:50.2429123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4ybyosp/_remote_module_non_scriptable.py 2022-09-27T16:00:50.2763932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8sveh7fw 2022-09-27T16:00:50.2765065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8sveh7fw/_remote_module_non_scriptable.py 2022-09-27T16:00:50.4947903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:50.4983500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:50.5008623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:50.5344299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:51.1761915Z ok (2.458s) 2022-09-27T16:00:51.1762171Z 2022-09-27T16:00:51.1762597Z ---------------------------------------------------------------------- 2022-09-27T16:00:51.1762997Z Ran 1 test in 2.458s 2022-09-27T16:00:51.1763168Z 2022-09-27T16:00:51.1763263Z OK 2022-09-27T16:00:51.1763402Z 2022-09-27T16:00:51.1763528Z Generating XML reports... 2022-09-27T16:00:51.1799288Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160048.xml 2022-09-27T16:00:52.5642995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:52.5643363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:52.5651203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:52.5651564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:52.7311396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6k6v0prh 2022-09-27T16:00:52.7312286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6k6v0prh/_remote_module_non_scriptable.py 2022-09-27T16:00:52.9950484Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:00:52.9960773Z 2022-09-27T16:00:52.9961051Z Running tests... 2022-09-27T16:00:52.9961522Z ---------------------------------------------------------------------- 2022-09-27T16:00:53.2360576Z test_py_rpc_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28214 2022-09-27T16:00:53.2387753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28215 2022-09-27T16:00:53.2415491Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28216 2022-09-27T16:00:53.2444641Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28217 2022-09-27T16:00:54.3944766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:54.3945451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:54.4279288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:54.4279922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:54.4512878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:54.4513546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:54.4520605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:54.4521205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:54.4651160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:54.4651682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:54.4885353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:54.4885897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:54.4895785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:54.4896418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:54.5021015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:54.5021644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:54.5542359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3s56arr4 2022-09-27T16:00:54.5543268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3s56arr4/_remote_module_non_scriptable.py 2022-09-27T16:00:54.6141617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvm0e5dj_ 2022-09-27T16:00:54.6142429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvm0e5dj_/_remote_module_non_scriptable.py 2022-09-27T16:00:54.6163653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxz2me2hz 2022-09-27T16:00:54.6165956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxz2me2hz/_remote_module_non_scriptable.py 2022-09-27T16:00:54.6273589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7pi3m90 2022-09-27T16:00:54.6275365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7pi3m90/_remote_module_non_scriptable.py 2022-09-27T16:00:54.8096647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:54.8707234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:54.8734687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:54.9021466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:55.6504315Z ok (2.654s) 2022-09-27T16:00:55.6504465Z 2022-09-27T16:00:55.6504780Z ---------------------------------------------------------------------- 2022-09-27T16:00:55.6505032Z Ran 1 test in 2.654s 2022-09-27T16:00:55.6505149Z 2022-09-27T16:00:55.6505197Z OK 2022-09-27T16:00:55.6505289Z 2022-09-27T16:00:55.6505382Z Generating XML reports... 2022-09-27T16:00:55.6543205Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160052.xml 2022-09-27T16:00:57.0265605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:57.0266010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:57.0273957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:57.0274316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:57.1881624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpagwdwg_7 2022-09-27T16:00:57.1882376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpagwdwg_7/_remote_module_non_scriptable.py 2022-09-27T16:00:57.4498218Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:00:57.4507334Z 2022-09-27T16:00:57.4507462Z Running tests... 2022-09-27T16:00:57.4507868Z ---------------------------------------------------------------------- 2022-09-27T16:00:57.7036559Z test_py_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28449 2022-09-27T16:00:57.7063327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28450 2022-09-27T16:00:57.7090456Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28451 2022-09-27T16:00:57.7118782Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28452 2022-09-27T16:00:58.7973475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:58.7974093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:58.8147441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:58.8148379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:58.8314697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:58.8315675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:58.8319922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:00:58.8320577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:00:58.8332156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:58.8332836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:58.8519044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:58.8519635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:58.8689226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:58.8689977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:58.8698385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:00:58.8699016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:00:58.9626884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjkr6f9b0 2022-09-27T16:00:58.9628317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjkr6f9b0/_remote_module_non_scriptable.py 2022-09-27T16:00:58.9812761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddh90v28 2022-09-27T16:00:58.9813732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddh90v28/_remote_module_non_scriptable.py 2022-09-27T16:00:58.9972977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0t_hp4u 2022-09-27T16:00:58.9974251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0t_hp4u/_remote_module_non_scriptable.py 2022-09-27T16:00:59.0131478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0djg6mjw 2022-09-27T16:00:59.0132920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0djg6mjw/_remote_module_non_scriptable.py 2022-09-27T16:00:59.2172942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:00:59.2361933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:00:59.2537799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:00:59.2692000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:00:59.9175437Z ok (2.466s) 2022-09-27T16:00:59.9175696Z 2022-09-27T16:00:59.9176053Z ---------------------------------------------------------------------- 2022-09-27T16:00:59.9176339Z Ran 1 test in 2.467s 2022-09-27T16:00:59.9176485Z 2022-09-27T16:00:59.9176578Z OK 2022-09-27T16:00:59.9176684Z 2022-09-27T16:00:59.9176781Z Generating XML reports... 2022-09-27T16:00:59.9212524Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160057.xml 2022-09-27T16:01:01.3301538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:01.3301919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:01.3311037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:01.3311393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:01.4956724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp911b6v8j 2022-09-27T16:01:01.4957512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp911b6v8j/_remote_module_non_scriptable.py 2022-09-27T16:01:01.7687305Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:01:01.7697628Z 2022-09-27T16:01:01.7697722Z Running tests... 2022-09-27T16:01:01.7698671Z ---------------------------------------------------------------------- 2022-09-27T16:01:02.0284719Z test_py_rref_args_user_share_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28684 2022-09-27T16:01:02.0314092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28685 2022-09-27T16:01:02.0344677Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28686 2022-09-27T16:01:02.0373726Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28687 2022-09-27T16:01:03.1200296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:03.1201043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:03.1489678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:03.1490283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:03.1539967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:03.1540706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:03.1555502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:03.1556078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:03.1611064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:03.1611714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:03.1870152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:03.1870820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:03.1915067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:03.1915707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:03.1982357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:03.1982991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:03.2819869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxg1_yml1 2022-09-27T16:01:03.2820648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxg1_yml1/_remote_module_non_scriptable.py 2022-09-27T16:01:03.3149705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdq7mfl2w 2022-09-27T16:01:03.3150865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdq7mfl2w/_remote_module_non_scriptable.py 2022-09-27T16:01:03.3176498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhdd7zce 2022-09-27T16:01:03.3177933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhdd7zce/_remote_module_non_scriptable.py 2022-09-27T16:01:03.3240765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbupc4xcj 2022-09-27T16:01:03.3242594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbupc4xcj/_remote_module_non_scriptable.py 2022-09-27T16:01:03.5399680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:01:03.5755075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:01:03.5758289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:01:03.5824194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:01:04.3431987Z ok (2.573s) 2022-09-27T16:01:04.3432242Z 2022-09-27T16:01:04.3432770Z ---------------------------------------------------------------------- 2022-09-27T16:01:04.3433032Z Ran 1 test in 2.573s 2022-09-27T16:01:04.3433150Z 2022-09-27T16:01:04.3433196Z OK 2022-09-27T16:01:04.3433292Z 2022-09-27T16:01:04.3433386Z Generating XML reports... 2022-09-27T16:01:04.3470542Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160101.xml 2022-09-27T16:01:05.7330271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:05.7330695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:05.7337692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:05.7338204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:05.8898514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ao9h7vs 2022-09-27T16:01:05.8899401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ao9h7vs/_remote_module_non_scriptable.py 2022-09-27T16:01:06.1504369Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:01:06.1515073Z 2022-09-27T16:01:06.1515225Z Running tests... 2022-09-27T16:01:06.1515806Z ---------------------------------------------------------------------- 2022-09-27T16:01:06.3979326Z test_py_sparse_tensors_in_container (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28919 2022-09-27T16:01:06.4006115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28920 2022-09-27T16:01:06.4033901Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28921 2022-09-27T16:01:06.4062452Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28922 2022-09-27T16:01:07.4728485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:07.4729207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:07.5071566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:07.5072151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:07.5160244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:07.5160902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:07.5232117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:07.5232735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:07.5294548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:07.5295185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:07.5528295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:07.5528968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:07.5609846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:07.5610803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:07.5666489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:07.5667415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:07.6347472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaab6gfzb 2022-09-27T16:01:07.6348309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaab6gfzb/_remote_module_non_scriptable.py 2022-09-27T16:01:07.6801815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9k1r_1j 2022-09-27T16:01:07.6802489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9k1r_1j/_remote_module_non_scriptable.py 2022-09-27T16:01:07.6889755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_n6x2132 2022-09-27T16:01:07.6891031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_n6x2132/_remote_module_non_scriptable.py 2022-09-27T16:01:07.6933261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tz7tq59 2022-09-27T16:01:07.6934251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tz7tq59/_remote_module_non_scriptable.py 2022-09-27T16:01:07.8905700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:01:07.9353574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:01:07.9486358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:01:07.9486931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:01:08.7121082Z ok (2.560s) 2022-09-27T16:01:08.7121252Z 2022-09-27T16:01:08.7121914Z ---------------------------------------------------------------------- 2022-09-27T16:01:08.7122208Z Ran 1 test in 2.561s 2022-09-27T16:01:08.7122326Z 2022-09-27T16:01:08.7122389Z OK 2022-09-27T16:01:08.7122491Z 2022-09-27T16:01:08.7122586Z Generating XML reports... 2022-09-27T16:01:08.7157658Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160106.xml 2022-09-27T16:01:10.0884189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:10.0884568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:10.0892790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:10.0893157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:10.2490976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxvp8wwv 2022-09-27T16:01:10.2491869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxvp8wwv/_remote_module_non_scriptable.py 2022-09-27T16:01:10.5150850Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:01:10.5160302Z 2022-09-27T16:01:10.5160398Z Running tests... 2022-09-27T16:01:10.5160789Z ---------------------------------------------------------------------- 2022-09-27T16:01:10.7605415Z test_rref_get_type_timeout_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29154 2022-09-27T16:01:10.7633064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29155 2022-09-27T16:01:10.7662091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29156 2022-09-27T16:01:10.7691589Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29157 2022-09-27T16:01:11.9185946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:11.9186910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:11.9272718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:11.9273628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:11.9399794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:11.9400490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:11.9425901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:11.9426565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:11.9593451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:11.9594034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:11.9686297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:11.9686957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:11.9813038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:11.9813664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:11.9838278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:11.9838923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:12.0860244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcj0kusl1 2022-09-27T16:01:12.0861054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcj0kusl1/_remote_module_non_scriptable.py 2022-09-27T16:01:12.0956260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7_rb79p 2022-09-27T16:01:12.0957529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7_rb79p/_remote_module_non_scriptable.py 2022-09-27T16:01:12.1072688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9v_62w9 2022-09-27T16:01:12.1073817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9v_62w9/_remote_module_non_scriptable.py 2022-09-27T16:01:12.1104435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5rxc3ax 2022-09-27T16:01:12.1106246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5rxc3ax/_remote_module_non_scriptable.py 2022-09-27T16:01:12.3434157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:01:12.3526566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:01:12.3646639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:01:12.3714463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:01:14.9779358Z ok (4.461s) 2022-09-27T16:01:14.9779643Z 2022-09-27T16:01:14.9780135Z ---------------------------------------------------------------------- 2022-09-27T16:01:14.9780388Z Ran 1 test in 4.462s 2022-09-27T16:01:14.9780492Z 2022-09-27T16:01:14.9780553Z OK 2022-09-27T16:01:14.9782361Z 2022-09-27T16:01:14.9782609Z Generating XML reports... 2022-09-27T16:01:14.9816279Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160110.xml 2022-09-27T16:01:16.3604074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:16.3604690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:16.3613342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:16.3614066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:16.5180378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0surpdk5 2022-09-27T16:01:16.5181042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0surpdk5/_remote_module_non_scriptable.py 2022-09-27T16:01:16.7826428Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:01:16.7836192Z 2022-09-27T16:01:16.7836662Z Running tests... 2022-09-27T16:01:16.7837076Z ---------------------------------------------------------------------- 2022-09-27T16:01:17.0258909Z test_rref_get_type_timeout_non_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29377 2022-09-27T16:01:17.0285438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29378 2022-09-27T16:01:17.0313327Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29379 2022-09-27T16:01:17.0341934Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29380 2022-09-27T16:01:18.1666967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:18.1667617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:18.1672907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:18.1673501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:18.2036625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:18.2037302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:18.2038148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:18.2038768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:18.2316829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:18.2317534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:18.2402097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:18.2402618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:18.2686784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:18.2687426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:18.2768280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:18.2768904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:18.3313881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjesauap 2022-09-27T16:01:18.3314789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0glu34u_ 2022-09-27T16:01:18.3315507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjesauap/_remote_module_non_scriptable.py 2022-09-27T16:01:18.3316218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0glu34u_/_remote_module_non_scriptable.py 2022-09-27T16:01:18.3940571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpel1avj 2022-09-27T16:01:18.3941875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpel1avj/_remote_module_non_scriptable.py 2022-09-27T16:01:18.4015756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr8ngq819 2022-09-27T16:01:18.4016511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr8ngq819/_remote_module_non_scriptable.py 2022-09-27T16:01:18.5872242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:01:18.5890061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:01:18.6496366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:01:18.6552433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:01:21.2430824Z ok (4.459s) 2022-09-27T16:01:21.2431163Z 2022-09-27T16:01:21.2431640Z ---------------------------------------------------------------------- 2022-09-27T16:01:21.2431897Z Ran 1 test in 4.459s 2022-09-27T16:01:21.2432016Z 2022-09-27T16:01:21.2432080Z OK 2022-09-27T16:01:21.2432172Z 2022-09-27T16:01:21.2432269Z Generating XML reports... 2022-09-27T16:01:21.2466665Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160116.xml 2022-09-27T16:01:22.6250178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:22.6250552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:22.6258526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:22.6258883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:22.7825310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbzawx_y 2022-09-27T16:01:22.7826720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbzawx_y/_remote_module_non_scriptable.py 2022-09-27T16:01:23.0469006Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:01:23.0479001Z 2022-09-27T16:01:23.0479128Z Running tests... 2022-09-27T16:01:23.0479559Z ---------------------------------------------------------------------- 2022-09-27T16:01:23.2944808Z test_rref_proxy_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29600 2022-09-27T16:01:23.2973668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29601 2022-09-27T16:01:23.3002404Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29602 2022-09-27T16:01:23.3032523Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29603 2022-09-27T16:01:24.4806458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:24.4807021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:24.4809843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:24.4810452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:24.4915527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:24.4916183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:24.5189474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:24.5190112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:24.5194980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:24.5195611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:24.5272803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:24.5273725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:24.5291005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:24.5291875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:24.5615650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:24.5616300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:24.6465320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6rmih2i 2022-09-27T16:01:24.6466135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6rmih2i/_remote_module_non_scriptable.py 2022-09-27T16:01:24.6477545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnaqzpu2 2022-09-27T16:01:24.6479107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnaqzpu2/_remote_module_non_scriptable.py 2022-09-27T16:01:24.6563541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpso_c9lj7 2022-09-27T16:01:24.6565165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpso_c9lj7/_remote_module_non_scriptable.py 2022-09-27T16:01:24.6893007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8pltd4mm 2022-09-27T16:01:24.6893817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8pltd4mm/_remote_module_non_scriptable.py 2022-09-27T16:01:24.9022655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:01:24.9048426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:01:24.9120460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:01:24.9457804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:01:37.2248025Z [W tensorpipe_agent.cpp:940] RPC agent for worker2 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T16:01:38.1724554Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #30 to worker0, as the agent is shutting down 2022-09-27T16:01:38.1907450Z [W tensorpipe_agent.cpp:625] RPC agent for worker0 won't send response to request #28 to worker3, as the agent is shutting down 2022-09-27T16:01:38.1945152Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #28 to worker2, as the agent is shutting down 2022-09-27T16:01:38.2043340Z [W tensorpipe_agent.cpp:625] RPC agent for worker2 won't send response to request #30 to worker1, as the agent is shutting down 2022-09-27T16:01:38.5289733Z ok (15.481s) 2022-09-27T16:01:38.5289978Z 2022-09-27T16:01:38.5290345Z ---------------------------------------------------------------------- 2022-09-27T16:01:38.5290603Z Ran 1 test in 15.481s 2022-09-27T16:01:38.5290719Z 2022-09-27T16:01:38.5290780Z OK 2022-09-27T16:01:38.5290865Z 2022-09-27T16:01:38.5291008Z Generating XML reports... 2022-09-27T16:01:38.5326566Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160123.xml 2022-09-27T16:01:39.9046828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:39.9047195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:39.9056713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:39.9057194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:40.0717956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67kwpvnk 2022-09-27T16:01:40.0719048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67kwpvnk/_remote_module_non_scriptable.py 2022-09-27T16:01:40.3308154Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:01:40.3317458Z 2022-09-27T16:01:40.3317594Z Running tests... 2022-09-27T16:01:40.3318182Z ---------------------------------------------------------------------- 2022-09-27T16:01:40.5771384Z test_self_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29823 2022-09-27T16:01:40.5798585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29824 2022-09-27T16:01:40.5826250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29825 2022-09-27T16:01:40.5855063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29826 2022-09-27T16:01:41.6989497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:41.6990135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:41.7017111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:41.7017742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:41.7357409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:41.7358156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:41.7383329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:41.7383912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:41.7495094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:41.7495713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:41.7857861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:41.7858529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:41.7873434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:41.7873984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:41.8235418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:41.8236160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:41.8636969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy98xajee 2022-09-27T16:01:41.8637814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy98xajee/_remote_module_non_scriptable.py 2022-09-27T16:01:41.8651970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgr4bvclt 2022-09-27T16:01:41.8653002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgr4bvclt/_remote_module_non_scriptable.py 2022-09-27T16:01:41.9121108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbtwy504q 2022-09-27T16:01:41.9121895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbtwy504q/_remote_module_non_scriptable.py 2022-09-27T16:01:41.9494098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzq99dv8_ 2022-09-27T16:01:41.9495169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzq99dv8_/_remote_module_non_scriptable.py 2022-09-27T16:01:42.1257604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:01:42.1423762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:01:42.1686490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:01:42.2070442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:01:42.8912970Z ok (2.559s) 2022-09-27T16:01:42.8913217Z 2022-09-27T16:01:42.8913651Z ---------------------------------------------------------------------- 2022-09-27T16:01:42.8914627Z Ran 1 test in 2.559s 2022-09-27T16:01:42.8914812Z 2022-09-27T16:01:42.8914882Z OK 2022-09-27T16:01:42.8914977Z 2022-09-27T16:01:42.8915076Z Generating XML reports... 2022-09-27T16:01:42.8948980Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160140.xml 2022-09-27T16:01:44.2666209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:44.2666600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:44.2673981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:44.2674337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:44.4218367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvagz4ij9 2022-09-27T16:01:44.4219889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvagz4ij9/_remote_module_non_scriptable.py 2022-09-27T16:01:44.6802202Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:01:44.6811395Z 2022-09-27T16:01:44.6811569Z Running tests... 2022-09-27T16:01:44.6812204Z ---------------------------------------------------------------------- 2022-09-27T16:01:44.9208213Z test_self_remote_rref_as_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30058 2022-09-27T16:01:44.9235207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30059 2022-09-27T16:01:44.9262711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30060 2022-09-27T16:01:44.9291708Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30061 2022-09-27T16:01:46.0463637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:46.0464326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:46.0831658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:46.0832367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:46.0842716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:46.0843345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:46.1226724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:46.1227334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:46.1737689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:46.1738378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:46.1745144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:46.1745784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:46.2109884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wdbkumo 2022-09-27T16:01:46.2111214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:46.2111852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:46.2112428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wdbkumo/_remote_module_non_scriptable.py 2022-09-27T16:01:46.2118840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:46.2119485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:46.2494263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9cor6c3 2022-09-27T16:01:46.2495460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9cor6c3/_remote_module_non_scriptable.py 2022-09-27T16:01:46.3374109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp540cf6j3 2022-09-27T16:01:46.3374783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbre6na3m 2022-09-27T16:01:46.3375390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp540cf6j3/_remote_module_non_scriptable.py 2022-09-27T16:01:46.3376050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbre6na3m/_remote_module_non_scriptable.py 2022-09-27T16:01:46.4685341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:01:46.5042580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:01:46.5904102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:01:46.5904725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:01:47.3353201Z ok (2.654s) 2022-09-27T16:01:47.3353476Z 2022-09-27T16:01:47.3353958Z ---------------------------------------------------------------------- 2022-09-27T16:01:47.3354210Z Ran 1 test in 2.654s 2022-09-27T16:01:47.3354331Z 2022-09-27T16:01:47.3354377Z OK 2022-09-27T16:01:47.3354471Z 2022-09-27T16:01:47.3354760Z Generating XML reports... 2022-09-27T16:01:47.3389270Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160144.xml 2022-09-27T16:01:48.6969974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:48.6970356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:48.6978016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:48.6978380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:48.8569183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7luih79y 2022-09-27T16:01:48.8570262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7luih79y/_remote_module_non_scriptable.py 2022-09-27T16:01:49.1181781Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:01:49.1191927Z 2022-09-27T16:01:49.1192336Z Running tests... 2022-09-27T16:01:49.1192743Z ---------------------------------------------------------------------- 2022-09-27T16:01:49.3636371Z test_self_remote_rref_as_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30293 2022-09-27T16:01:49.3663740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30294 2022-09-27T16:01:49.3691650Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30295 2022-09-27T16:01:49.3721507Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30296 2022-09-27T16:01:50.4864449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:50.4865129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:50.4877922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:50.4878663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:50.5026689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:50.5027228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:50.5240785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:50.5241403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:50.5258441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:50.5258974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:50.5278195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:50.5278706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:50.5401408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:50.5402041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:50.5639625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:50.5640299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:50.6508564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58cu7w2t 2022-09-27T16:01:50.6509369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58cu7w2t/_remote_module_non_scriptable.py 2022-09-27T16:01:50.6518367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5nmtz4y7 2022-09-27T16:01:50.6520002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5nmtz4y7/_remote_module_non_scriptable.py 2022-09-27T16:01:50.6670223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyh0oia3c 2022-09-27T16:01:50.6671235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyh0oia3c/_remote_module_non_scriptable.py 2022-09-27T16:01:50.6913445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_siga1u7 2022-09-27T16:01:50.6914190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_siga1u7/_remote_module_non_scriptable.py 2022-09-27T16:01:50.9105148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:01:50.9105680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:01:50.9263182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:01:50.9527403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:01:51.6779500Z ok (2.558s) 2022-09-27T16:01:51.6779676Z 2022-09-27T16:01:51.6780041Z ---------------------------------------------------------------------- 2022-09-27T16:01:51.6780303Z Ran 1 test in 2.559s 2022-09-27T16:01:51.6780410Z 2022-09-27T16:01:51.6781088Z OK 2022-09-27T16:01:51.6781232Z 2022-09-27T16:01:51.6781351Z Generating XML reports... 2022-09-27T16:01:51.6817714Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160149.xml 2022-09-27T16:01:53.0174677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:53.0175040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:53.0183732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:53.0184102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:53.1771415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazmvgsap 2022-09-27T16:01:53.1772268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazmvgsap/_remote_module_non_scriptable.py 2022-09-27T16:01:53.4373066Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:01:53.4383076Z 2022-09-27T16:01:53.4383589Z Running tests... 2022-09-27T16:01:53.4383982Z ---------------------------------------------------------------------- 2022-09-27T16:01:53.6763361Z test_self_remote_rref_as_self_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30528 2022-09-27T16:01:53.6790236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30529 2022-09-27T16:01:53.6818340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30530 2022-09-27T16:01:53.6846965Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30531 2022-09-27T16:01:54.8044563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:54.8045201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:54.8063223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:54.8063862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:54.8130762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:54.8131421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:54.8416054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:54.8416713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:54.8439379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:54.8440107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:54.8499184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:54.8499702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:54.8500461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:54.8501027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:54.8854365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:54.8855029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:54.9678817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpokoqd91_ 2022-09-27T16:01:54.9679784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpokoqd91_/_remote_module_non_scriptable.py 2022-09-27T16:01:54.9699224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkyzqjltz 2022-09-27T16:01:54.9700744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkyzqjltz/_remote_module_non_scriptable.py 2022-09-27T16:01:54.9759126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vkfngdx 2022-09-27T16:01:54.9761528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vkfngdx/_remote_module_non_scriptable.py 2022-09-27T16:01:55.0114864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8v206v9g 2022-09-27T16:01:55.0115935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8v206v9g/_remote_module_non_scriptable.py 2022-09-27T16:01:55.2235361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:01:55.2255251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:01:55.2331494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:01:55.2695307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:01:55.9905310Z ok (2.552s) 2022-09-27T16:01:55.9905580Z 2022-09-27T16:01:55.9906062Z ---------------------------------------------------------------------- 2022-09-27T16:01:55.9906332Z Ran 1 test in 2.552s 2022-09-27T16:01:55.9906438Z 2022-09-27T16:01:55.9906500Z OK 2022-09-27T16:01:55.9906594Z 2022-09-27T16:01:55.9906690Z Generating XML reports... 2022-09-27T16:01:55.9943584Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160153.xml 2022-09-27T16:01:57.3909945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:57.3910564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:57.3919188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:57.3919575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:57.5495681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wx2chw7 2022-09-27T16:01:57.5496434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wx2chw7/_remote_module_non_scriptable.py 2022-09-27T16:01:57.8084333Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:01:57.8093366Z 2022-09-27T16:01:57.8093574Z Running tests... 2022-09-27T16:01:57.8094221Z ---------------------------------------------------------------------- 2022-09-27T16:01:58.0490055Z test_self_remote_rref_as_self_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30763 2022-09-27T16:01:58.0516675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30764 2022-09-27T16:01:58.0544482Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30765 2022-09-27T16:01:58.0574156Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30766 2022-09-27T16:01:59.1502823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:59.1503491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:59.1789116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:59.1789795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:59.1837086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:59.1837816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:59.1885601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:59.1886160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:59.1912994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:01:59.1913533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:01:59.2188650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:59.2232362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:59.2233229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:59.2233889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:59.2307591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:01:59.3160257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:01:59.3160927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp437sapi1 2022-09-27T16:01:59.3161681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp437sapi1/_remote_module_non_scriptable.py 2022-09-27T16:01:59.3457838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdbmspbgu 2022-09-27T16:01:59.3458604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdbmspbgu/_remote_module_non_scriptable.py 2022-09-27T16:01:59.3495750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3zfq4aeb 2022-09-27T16:01:59.3496558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3zfq4aeb/_remote_module_non_scriptable.py 2022-09-27T16:01:59.3580378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0bmhahk 2022-09-27T16:01:59.3581795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0bmhahk/_remote_module_non_scriptable.py 2022-09-27T16:01:59.5757979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:01:59.6082681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:01:59.6192313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:01:59.6289316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:02:00.3632034Z ok (2.553s) 2022-09-27T16:02:00.3632256Z 2022-09-27T16:02:00.3632764Z ---------------------------------------------------------------------- 2022-09-27T16:02:00.3633208Z Ran 1 test in 2.554s 2022-09-27T16:02:00.3633385Z 2022-09-27T16:02:00.3633458Z OK 2022-09-27T16:02:00.3633594Z 2022-09-27T16:02:00.3637046Z Generating XML reports... 2022-09-27T16:02:00.3668733Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160157.xml 2022-09-27T16:02:01.7446931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:01.7447300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:01.7455872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:01.7456246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:01.9069755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1ekzcy3 2022-09-27T16:02:01.9071237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1ekzcy3/_remote_module_non_scriptable.py 2022-09-27T16:02:02.1728476Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:02:02.1738362Z 2022-09-27T16:02:02.1738785Z Running tests... 2022-09-27T16:02:02.1739185Z ---------------------------------------------------------------------- 2022-09-27T16:02:02.4183423Z test_send_to_rank_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30998 2022-09-27T16:02:02.4213550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30999 2022-09-27T16:02:02.4241859Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31000 2022-09-27T16:02:02.4273332Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31001 2022-09-27T16:02:03.5731430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:03.5732156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:03.5987878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:03.5988512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:03.6090125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:03.6090688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:03.6193196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:03.6363168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:03.6364158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:03.6364752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:03.6430803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:03.6431629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:03.6574554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:03.6575197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:03.6933881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:03.6934562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:03.7397989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpefcma47c 2022-09-27T16:02:03.7399259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpefcma47c/_remote_module_non_scriptable.py 2022-09-27T16:02:03.7673497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vp0b0vi 2022-09-27T16:02:03.7674276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vp0b0vi/_remote_module_non_scriptable.py 2022-09-27T16:02:03.7885416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_epb7mz 2022-09-27T16:02:03.7886202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_epb7mz/_remote_module_non_scriptable.py 2022-09-27T16:02:03.8199585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmsh6grkl 2022-09-27T16:02:03.8200490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmsh6grkl/_remote_module_non_scriptable.py 2022-09-27T16:02:03.9957416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:02:04.0219096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:02:04.0458464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:02:04.0754798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:02:04.8331871Z ok (2.659s) 2022-09-27T16:02:04.8332146Z 2022-09-27T16:02:04.8332600Z ---------------------------------------------------------------------- 2022-09-27T16:02:04.8333032Z Ran 1 test in 2.659s 2022-09-27T16:02:04.8333246Z 2022-09-27T16:02:04.8333308Z OK 2022-09-27T16:02:04.8333401Z 2022-09-27T16:02:04.8333497Z Generating XML reports... 2022-09-27T16:02:04.8368177Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160202.xml 2022-09-27T16:02:06.1959854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:06.1960248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:06.1967373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:06.1967737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:06.3522980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvr_pt6gs 2022-09-27T16:02:06.3523994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvr_pt6gs/_remote_module_non_scriptable.py 2022-09-27T16:02:06.6137561Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:02:06.6147271Z 2022-09-27T16:02:06.6147509Z Running tests... 2022-09-27T16:02:06.6147834Z ---------------------------------------------------------------------- 2022-09-27T16:02:06.8632978Z test_set_and_get_num_worker_threads (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31233 2022-09-27T16:02:06.8659469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31234 2022-09-27T16:02:06.8687412Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31235 2022-09-27T16:02:06.8716273Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31236 2022-09-27T16:02:07.9729145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:07.9729811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:07.9812972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:07.9813637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:08.0083056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:08.0083687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:08.0092263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:08.0092910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:08.0145861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:08.0146355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:08.0188371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:08.0188918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:08.0459012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:08.0459661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:08.0514952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:08.0515650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:08.1351754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nmqmp0c 2022-09-27T16:02:08.1352545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nmqmp0c/_remote_module_non_scriptable.py 2022-09-27T16:02:08.1451254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9exj7513 2022-09-27T16:02:08.1452751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9exj7513/_remote_module_non_scriptable.py 2022-09-27T16:02:08.1722232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph87nouzk 2022-09-27T16:02:08.1723746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph87nouzk/_remote_module_non_scriptable.py 2022-09-27T16:02:08.1767480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplntbqpff 2022-09-27T16:02:08.1769229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplntbqpff/_remote_module_non_scriptable.py 2022-09-27T16:02:08.3912958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:02:08.4014683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:02:08.4312206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:02:08.4330532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:02:09.0772565Z ok (2.462s) 2022-09-27T16:02:09.0772804Z 2022-09-27T16:02:09.0773319Z ---------------------------------------------------------------------- 2022-09-27T16:02:09.0773651Z Ran 1 test in 2.462s 2022-09-27T16:02:09.0773769Z 2022-09-27T16:02:09.0773844Z OK 2022-09-27T16:02:09.0773937Z 2022-09-27T16:02:09.0774018Z Generating XML reports... 2022-09-27T16:02:09.0809466Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160206.xml 2022-09-27T16:02:10.4530209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:10.4530586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:10.4538819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:10.4539177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:10.6127058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8ux3bg9 2022-09-27T16:02:10.6128512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8ux3bg9/_remote_module_non_scriptable.py 2022-09-27T16:02:10.8749930Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:02:10.8759646Z 2022-09-27T16:02:10.8759802Z Running tests... 2022-09-27T16:02:10.8760141Z ---------------------------------------------------------------------- 2022-09-27T16:02:11.1241248Z test_stress_heavy_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31500 2022-09-27T16:02:11.1268138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31501 2022-09-27T16:02:11.1295696Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31502 2022-09-27T16:02:11.1324854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31503 2022-09-27T16:02:12.2286686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:12.2287348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:12.2590381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:12.2591090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:12.2613301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:12.2613939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:12.2621295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:12.2621944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:12.2671816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:12.2672499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:12.2976343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:12.2977192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:12.3003667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:12.3004300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:12.3012809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:12.3013420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:12.3941229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvu5k_ky 2022-09-27T16:02:12.3941901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvu5k_ky/_remote_module_non_scriptable.py 2022-09-27T16:02:12.4233678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwej7yzmv 2022-09-27T16:02:12.4234484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwej7yzmv/_remote_module_non_scriptable.py 2022-09-27T16:02:12.4276867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5pdz61u 2022-09-27T16:02:12.4277547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5pdz61u/_remote_module_non_scriptable.py 2022-09-27T16:02:12.4280081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjdl3_53y 2022-09-27T16:02:12.4282519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjdl3_53y/_remote_module_non_scriptable.py 2022-09-27T16:02:12.6539091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:02:12.6842602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:02:12.6843131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:02:12.6892926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:02:13.0005979Z Rank 0 finished testing 20 times in 0.08450651168823242 seconds. 2022-09-27T16:02:13.0432329Z Rank 1 finished testing 20 times in 0.12144780158996582 seconds. 2022-09-27T16:02:13.0538576Z Rank 3 finished testing 20 times in 0.09705328941345215 seconds. 2022-09-27T16:02:13.0636531Z Rank 2 finished testing 20 times in 0.103424072265625 seconds. 2022-09-27T16:02:13.4383464Z ok (2.562s) 2022-09-27T16:02:13.4383648Z 2022-09-27T16:02:13.4383963Z ---------------------------------------------------------------------- 2022-09-27T16:02:13.4384272Z Ran 1 test in 2.562s 2022-09-27T16:02:13.4384387Z 2022-09-27T16:02:13.4384449Z OK 2022-09-27T16:02:13.4384527Z 2022-09-27T16:02:13.4384619Z Generating XML reports... 2022-09-27T16:02:13.4420252Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160210.xml 2022-09-27T16:02:14.8260655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:14.8261109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:14.8268973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:14.8269341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:14.9824738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwctgml5q 2022-09-27T16:02:14.9825782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwctgml5q/_remote_module_non_scriptable.py 2022-09-27T16:02:15.2466969Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:02:15.2476590Z 2022-09-27T16:02:15.2476908Z Running tests... 2022-09-27T16:02:15.2477807Z ---------------------------------------------------------------------- 2022-09-27T16:02:15.5006114Z test_tensorpipe_options_throw_on_timedelta_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31735 2022-09-27T16:02:15.5033997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31736 2022-09-27T16:02:15.5061746Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31737 2022-09-27T16:02:15.5091227Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31738 2022-09-27T16:02:16.6394477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:16.6395329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:16.6490648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:16.6491289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:16.6675388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:16.6676081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:16.6779412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:16.6779948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:16.6780964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:16.6781446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:16.6879478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:16.6880140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:16.7068882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:16.7069578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:16.7162299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:16.7162923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:16.8049577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbf29uwdc 2022-09-27T16:02:16.8050400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbf29uwdc/_remote_module_non_scriptable.py 2022-09-27T16:02:16.8144098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphrbne58 2022-09-27T16:02:16.8145468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphrbne58/_remote_module_non_scriptable.py 2022-09-27T16:02:16.8332219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2y9ttfn 2022-09-27T16:02:16.8333037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2y9ttfn/_remote_module_non_scriptable.py 2022-09-27T16:02:16.8416474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpub096bdi 2022-09-27T16:02:16.8417615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpub096bdi/_remote_module_non_scriptable.py 2022-09-27T16:02:17.0635498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:02:17.0727376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:02:17.0935169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:02:17.0989894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:02:17.4140761Z ok (2.166s) 2022-09-27T16:02:17.4141231Z 2022-09-27T16:02:17.4141570Z ---------------------------------------------------------------------- 2022-09-27T16:02:17.4141918Z Ran 1 test in 2.166s 2022-09-27T16:02:17.4142068Z 2022-09-27T16:02:17.4142131Z OK 2022-09-27T16:02:17.4142226Z 2022-09-27T16:02:17.4142319Z Generating XML reports... 2022-09-27T16:02:17.4177619Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160215.xml 2022-09-27T16:02:18.7860372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:18.7860762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:18.7868519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:18.7868946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:18.9449320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbji8neu 2022-09-27T16:02:18.9450969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbji8neu/_remote_module_non_scriptable.py 2022-09-27T16:02:19.2070669Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:02:19.2080592Z 2022-09-27T16:02:19.2080744Z Running tests... 2022-09-27T16:02:19.2081385Z ---------------------------------------------------------------------- 2022-09-27T16:02:19.4498244Z test_tensorpipe_set_default_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31794 2022-09-27T16:02:19.4526087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31795 2022-09-27T16:02:19.4553696Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31796 2022-09-27T16:02:19.4582917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31797 2022-09-27T16:02:20.6414141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:20.6414826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:20.6666517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:20.6668956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:20.6670005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:20.6670616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:20.6795576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:20.6796158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:20.7059825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:20.7060430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:20.7061197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:20.7061749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:20.7289375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:20.7290030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:20.7662481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:20.7663163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:20.8068757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9zg1qyr 2022-09-27T16:02:20.8070153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9zg1qyr/_remote_module_non_scriptable.py 2022-09-27T16:02:20.8339201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9tfvce9d 2022-09-27T16:02:20.8339739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9tfvce9d/_remote_module_non_scriptable.py 2022-09-27T16:02:20.8358357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97o47dkv 2022-09-27T16:02:20.8361324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97o47dkv/_remote_module_non_scriptable.py 2022-09-27T16:02:20.9041582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwehesn6 2022-09-27T16:02:20.9042440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwehesn6/_remote_module_non_scriptable.py 2022-09-27T16:02:21.0665123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:02:21.0962019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:02:21.1075218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:02:21.1658438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:02:21.8645062Z ok (2.656s) 2022-09-27T16:02:21.8646883Z 2022-09-27T16:02:21.8647855Z ---------------------------------------------------------------------- 2022-09-27T16:02:21.8648391Z Ran 1 test in 2.656s 2022-09-27T16:02:21.8648579Z 2022-09-27T16:02:21.8648656Z OK 2022-09-27T16:02:21.8648877Z 2022-09-27T16:02:21.8649013Z Generating XML reports... 2022-09-27T16:02:21.8689362Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160219.xml 2022-09-27T16:02:23.2733309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:23.2733682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:23.2741740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:23.2742099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:23.4343364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrs58jlc 2022-09-27T16:02:23.4344492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrs58jlc/_remote_module_non_scriptable.py 2022-09-27T16:02:23.6996738Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:02:23.7006519Z 2022-09-27T16:02:23.7006609Z Running tests... 2022-09-27T16:02:23.7007766Z ---------------------------------------------------------------------- 2022-09-27T16:02:23.9409064Z test_wait_all_workers_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32017 2022-09-27T16:02:23.9436124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32018 2022-09-27T16:02:23.9464131Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32019 2022-09-27T16:02:23.9494421Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32020 2022-09-27T16:02:25.0999436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:25.1000082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:25.1360108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:25.1360799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:25.1361879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:25.1362531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:25.1388695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:25.1389353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:25.1546380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:25.1546906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:25.1728828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:25.1729463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:25.1769538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:25.1770195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:25.1919007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:25.1919655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:25.2630711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegy46yx2 2022-09-27T16:02:25.2631603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegy46yx2/_remote_module_non_scriptable.py 2022-09-27T16:02:25.2998266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmkpp80l 2022-09-27T16:02:25.2999033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmkpp80l/_remote_module_non_scriptable.py 2022-09-27T16:02:25.3024223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3wbgegxs 2022-09-27T16:02:25.3025312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3wbgegxs/_remote_module_non_scriptable.py 2022-09-27T16:02:25.3184341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgfbsbso 2022-09-27T16:02:25.3185137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgfbsbso/_remote_module_non_scriptable.py 2022-09-27T16:02:25.5437304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:02:25.5612532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:02:25.5641537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:02:25.5807997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:02:25.6022741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:02:25.6023141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:02:25.6124470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:02:25.6126428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:02:25.6127084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:02:25.6127617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:02:25.6128169Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:02:25.6128950Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:02:26.0675442Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T16:02:26.0676372Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T16:02:26.0677681Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T16:02:26.0678918Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-09-27T16:02:26.0680329Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T16:02:26.0681934Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T16:02:26.4556182Z ok (2.755s) 2022-09-27T16:02:26.4556411Z 2022-09-27T16:02:26.4556927Z ---------------------------------------------------------------------- 2022-09-27T16:02:26.4557316Z Ran 1 test in 2.755s 2022-09-27T16:02:26.4557431Z 2022-09-27T16:02:26.4557479Z OK 2022-09-27T16:02:26.4557568Z 2022-09-27T16:02:26.4557660Z Generating XML reports... 2022-09-27T16:02:26.4592251Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160223.xml 2022-09-27T16:02:27.8480703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:27.8481086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:27.8488688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:27.8489229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:28.0041852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnzgina0 2022-09-27T16:02:28.0042732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnzgina0/_remote_module_non_scriptable.py 2022-09-27T16:02:28.2634188Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:02:28.2643894Z 2022-09-27T16:02:28.2644195Z Running tests... 2022-09-27T16:02:28.2644894Z ---------------------------------------------------------------------- 2022-09-27T16:02:28.5041673Z test_wait_all_workers_twice_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32264 2022-09-27T16:02:28.5069230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32265 2022-09-27T16:02:28.5096092Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32266 2022-09-27T16:02:28.5125165Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32267 2022-09-27T16:02:29.5655846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:29.5656468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:29.5986719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:29.5987275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:29.6008160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:29.6008794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:29.6154819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:29.6375971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:29.6377400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:29.6377987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:29.6378707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:29.6379252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:29.6406661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:29.6748743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:29.6749694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:29.6750299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:29.7412337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2_fahux 2022-09-27T16:02:29.7413107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2_fahux/_remote_module_non_scriptable.py 2022-09-27T16:02:29.7633034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dyjeuzh 2022-09-27T16:02:29.7633789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dyjeuzh/_remote_module_non_scriptable.py 2022-09-27T16:02:29.7641181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsg589641 2022-09-27T16:02:29.7642849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsg589641/_remote_module_non_scriptable.py 2022-09-27T16:02:29.8008715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbfypze_x 2022-09-27T16:02:29.8009650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbfypze_x/_remote_module_non_scriptable.py 2022-09-27T16:02:29.9924351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:02:30.0156413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:02:30.0158153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:02:30.0559550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:02:30.0740766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:02:30.0942719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:02:30.1045449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:02:30.1046186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:02:30.1047244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:02:30.1048090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:02:30.1048934Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:02:30.1049752Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:02:30.6046237Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T16:02:30.6047418Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T16:02:30.6050279Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T16:02:30.6053147Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T16:02:30.6054078Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T16:02:30.6054948Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T16:02:31.0186890Z ok (2.754s) 2022-09-27T16:02:31.0187119Z 2022-09-27T16:02:31.0187591Z ---------------------------------------------------------------------- 2022-09-27T16:02:31.0187973Z Ran 1 test in 2.754s 2022-09-27T16:02:31.0188171Z 2022-09-27T16:02:31.0188256Z OK 2022-09-27T16:02:31.0188388Z 2022-09-27T16:02:31.0188524Z Generating XML reports... 2022-09-27T16:02:31.0224843Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160228.xml 2022-09-27T16:02:32.3892457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:32.3892848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:32.3899780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:32.3900141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:32.5475423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zivevjx 2022-09-27T16:02:32.5476108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zivevjx/_remote_module_non_scriptable.py 2022-09-27T16:02:32.8083349Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:02:32.8093536Z 2022-09-27T16:02:32.8094007Z Running tests... 2022-09-27T16:02:32.8094467Z ---------------------------------------------------------------------- 2022-09-27T16:02:33.0544302Z test_world_size_one_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32511 2022-09-27T16:02:33.0571886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32512 2022-09-27T16:02:33.0600112Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32513 2022-09-27T16:02:33.0629849Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32514 2022-09-27T16:02:34.1450012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:34.1450633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:34.1465103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:34.1465796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:34.1810237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:34.1810957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:34.1816290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:34.1816877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:34.1836079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:34.1836687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:34.1938027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:34.1938670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:34.2181430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:34.2182093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:34.2308435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:34.2309063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:34.3085594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt24d5o6 2022-09-27T16:02:34.3087014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt24d5o6/_remote_module_non_scriptable.py 2022-09-27T16:02:34.3106850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ocm022t 2022-09-27T16:02:34.3108388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ocm022t/_remote_module_non_scriptable.py 2022-09-27T16:02:34.3441951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qhkp5o9 2022-09-27T16:02:34.3443109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qhkp5o9/_remote_module_non_scriptable.py 2022-09-27T16:02:34.3587919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_c3foco 2022-09-27T16:02:34.3588766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_c3foco/_remote_module_non_scriptable.py 2022-09-27T16:02:34.5640434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:02:34.5684862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:02:34.6013957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:02:34.6194371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:02:34.9680711Z ok (2.158s) 2022-09-27T16:02:34.9680924Z 2022-09-27T16:02:34.9681357Z ---------------------------------------------------------------------- 2022-09-27T16:02:34.9681744Z Ran 1 test in 2.159s 2022-09-27T16:02:34.9681930Z 2022-09-27T16:02:34.9682021Z OK 2022-09-27T16:02:34.9682163Z 2022-09-27T16:02:34.9682308Z Generating XML reports... 2022-09-27T16:02:34.9718039Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160232.xml 2022-09-27T16:02:36.3274795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:36.3275208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:36.3283522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:36.3283937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:36.4875259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zxl7x6i 2022-09-27T16:02:36.4876537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zxl7x6i/_remote_module_non_scriptable.py 2022-09-27T16:02:36.7510163Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:02:36.7519413Z 2022-09-27T16:02:36.7519533Z Running tests... 2022-09-27T16:02:36.7520104Z ---------------------------------------------------------------------- 2022-09-27T16:02:36.9969566Z test_create_remote_module_from_module_rref (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32620 2022-09-27T16:02:36.9996977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32621 2022-09-27T16:02:37.0024961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32622 2022-09-27T16:02:38.0390152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:38.0390899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:38.0719224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:38.0719875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:38.0955649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:38.0956027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:38.1046211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:38.1046904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:38.1327901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:38.1328512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:38.1413139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:38.1413779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:38.1976045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhouy4mg 2022-09-27T16:02:38.1977210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhouy4mg/_remote_module_non_scriptable.py 2022-09-27T16:02:38.2585488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1245fkk 2022-09-27T16:02:38.2585954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1245fkk/_remote_module_non_scriptable.py 2022-09-27T16:02:38.2664047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt68s4014 2022-09-27T16:02:38.2665479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt68s4014/_remote_module_non_scriptable.py 2022-09-27T16:02:38.4493224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:02:38.5141353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:02:38.5182692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:02:39.1073740Z ok (2.355s) 2022-09-27T16:02:39.1074006Z 2022-09-27T16:02:39.1074443Z ---------------------------------------------------------------------- 2022-09-27T16:02:39.1075004Z Ran 1 test in 2.355s 2022-09-27T16:02:39.1075196Z 2022-09-27T16:02:39.1075288Z OK 2022-09-27T16:02:39.1075429Z 2022-09-27T16:02:39.1075556Z Generating XML reports... 2022-09-27T16:02:39.1112582Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220927160236.xml 2022-09-27T16:02:40.4774392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:40.4774764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:40.4783088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:40.4783447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:40.6372739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz57h1zes 2022-09-27T16:02:40.6373832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz57h1zes/_remote_module_non_scriptable.py 2022-09-27T16:02:40.8979601Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:02:40.8988702Z 2022-09-27T16:02:40.8988849Z Running tests... 2022-09-27T16:02:40.8989480Z ---------------------------------------------------------------------- 2022-09-27T16:02:41.1433305Z test_send_remote_module_over_the_wire (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 322 2022-09-27T16:02:41.1459928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 323 2022-09-27T16:02:41.1488028Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 324 2022-09-27T16:02:42.1949580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:42.1950262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:42.2099310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:42.2099877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:42.2303076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:42.2303699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:42.2463525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:42.2464191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:42.2471456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:42.2472050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:42.2836841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:42.2837503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:42.3578149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4nv9m00 2022-09-27T16:02:42.3578826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4nv9m00/_remote_module_non_scriptable.py 2022-09-27T16:02:42.3731445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpg9yxpx 2022-09-27T16:02:42.3733293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpg9yxpx/_remote_module_non_scriptable.py 2022-09-27T16:02:42.4090831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8nzqonj9 2022-09-27T16:02:42.4098711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8nzqonj9/_remote_module_non_scriptable.py 2022-09-27T16:02:42.6132868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:02:42.6286534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:02:42.6626466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:02:43.2539653Z ok (2.355s) 2022-09-27T16:02:43.2539899Z 2022-09-27T16:02:43.2540237Z ---------------------------------------------------------------------- 2022-09-27T16:02:43.2540478Z Ran 1 test in 2.355s 2022-09-27T16:02:43.2540598Z 2022-09-27T16:02:43.2540659Z OK 2022-09-27T16:02:43.2540753Z 2022-09-27T16:02:43.2540847Z Generating XML reports... 2022-09-27T16:02:43.2576874Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220927160240.xml 2022-09-27T16:02:44.6279323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:44.6279931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:44.6287843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:44.6288427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:44.7872771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58opfmfi 2022-09-27T16:02:44.7873703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58opfmfi/_remote_module_non_scriptable.py 2022-09-27T16:02:45.0486078Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-09-27T16:02:45.0495431Z 2022-09-27T16:02:45.0495624Z Running tests... 2022-09-27T16:02:45.0496204Z ---------------------------------------------------------------------- 2022-09-27T16:02:45.2936515Z test_send_remote_module_over_the_wire_script_not_supported (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 492 2022-09-27T16:02:45.2963371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 493 2022-09-27T16:02:45.2990920Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 494 2022-09-27T16:02:46.4309111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:46.4309875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:46.4391121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:46.4391760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:46.4588135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:46.4588826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:46.4687652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:46.4688185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:46.4770572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:46.4771097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:46.4964934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:46.4965545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:46.5944068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2sag3u72 2022-09-27T16:02:46.5945200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2sag3u72/_remote_module_non_scriptable.py 2022-09-27T16:02:46.6028987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgpeb4yxc 2022-09-27T16:02:46.6031187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgpeb4yxc/_remote_module_non_scriptable.py 2022-09-27T16:02:46.6215007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeaey5w7v 2022-09-27T16:02:46.6215792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeaey5w7v/_remote_module_non_scriptable.py 2022-09-27T16:02:46.8476042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:02:46.8569259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:02:46.8742883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:02:47.0332133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgpeb4yxc/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T16:02:47.0333682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeaey5w7v/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T16:02:47.0404236Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpeaey5w7v/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-09-27T16:02:47.4043709Z ok (2.354s) 2022-09-27T16:02:47.4043969Z 2022-09-27T16:02:47.4044490Z ---------------------------------------------------------------------- 2022-09-27T16:02:47.4044859Z Ran 1 test in 2.355s 2022-09-27T16:02:47.4044961Z 2022-09-27T16:02:47.4045024Z OK 2022-09-27T16:02:47.4045116Z 2022-09-27T16:02:47.4045211Z Generating XML reports... 2022-09-27T16:02:47.4079307Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220927160245.xml 2022-09-27T16:02:48.0112438Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2022-09-27 16:02:48.010815] 2022-09-27T16:02:48.0113348Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:02:48.010905] 2022-09-27T16:02:49.2201519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2vn4sxi 2022-09-27T16:02:49.2202313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2vn4sxi/_remote_module_non_scriptable.py 2022-09-27T16:02:49.4796744Z ]> 2022-09-27T16:02:49.4797486Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) 2022-09-27T16:02:49.4798507Z , <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]> 2022-09-27T16:02:49.4799386Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) 2022-09-27T16:02:49.4799874Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) 2022-09-27T16:02:49.4800377Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) 2022-09-27T16:02:49.4801534Z , <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]> 2022-09-27T16:02:49.4802937Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) 2022-09-27T16:02:49.4803443Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) 2022-09-27T16:02:49.4804097Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) 2022-09-27T16:02:49.4804741Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) 2022-09-27T16:02:49.4805527Z ]> 2022-09-27T16:02:49.4806258Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) 2022-09-27T16:02:49.4808719Z , <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]> 2022-09-27T16:02:49.4810798Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) 2022-09-27T16:02:49.4811574Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) 2022-09-27T16:02:49.4812222Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) 2022-09-27T16:02:49.4812901Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) 2022-09-27T16:02:49.4813618Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) 2022-09-27T16:02:49.4814286Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) 2022-09-27T16:02:49.4814924Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) 2022-09-27T16:02:49.4815601Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) 2022-09-27T16:02:49.4841104Z , <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]> 2022-09-27T16:02:49.4861666Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4862048Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4862437Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4862834Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4863226Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4863620Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4864047Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4864466Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4864862Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4865248Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4865615Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4865973Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4866376Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4866736Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4867110Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4867481Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4867819Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4868159Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4868509Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4868866Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4869205Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4869563Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4869938Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4870289Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4870661Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4871123Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4871475Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4871863Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4872217Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4872605Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4872936Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4873285Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4873633Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4873988Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4874336Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4874839Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4875199Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4875544Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4875900Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4876257Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4876700Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4877046Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4877425Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4877804Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4878165Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4878525Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4878875Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4879249Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4879623Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4880001Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4880361Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4880712Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4881084Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4881467Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4881854Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4882231Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4882622Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4883019Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4883378Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4883738Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4884097Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4884448Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4884789Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4885155Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4885601Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4885948Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4886348Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4886717Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4887106Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4887487Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4887878Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4888266Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4888630Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4889004Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4889372Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4889738Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4890087Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4890463Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4890839Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4891202Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4891576Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4891951Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4892413Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4892767Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4893140Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4893518Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4893877Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-09-27T16:02:49.4894571Z , <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]> 2022-09-27T16:02:49.4895256Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2022-09-27T16:02:49.4895645Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2022-09-27T16:02:49.4896022Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2022-09-27T16:02:50.5202604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:50.5203338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:50.5213933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:50.5214502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:50.6830196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkyn8lfb2 2022-09-27T16:02:50.6831010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkyn8lfb2/_remote_module_non_scriptable.py 2022-09-27T16:02:50.9417172Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:02:50.9427864Z 2022-09-27T16:02:50.9428232Z Running tests... 2022-09-27T16:02:50.9428818Z ---------------------------------------------------------------------- 2022-09-27T16:02:50.9451014Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:02:50.9451552Z 2022-09-27T16:02:50.9451854Z ---------------------------------------------------------------------- 2022-09-27T16:02:50.9452105Z Ran 1 test in 0.002s 2022-09-27T16:02:50.9452208Z 2022-09-27T16:02:50.9452283Z OK (skipped=1) 2022-09-27T16:02:50.9452396Z 2022-09-27T16:02:50.9452484Z Generating XML reports... 2022-09-27T16:02:50.9482059Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20220927160250.xml 2022-09-27T16:02:52.2036850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:52.2037234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:52.2045564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:52.2045971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:52.3631182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ujnvwid 2022-09-27T16:02:52.3632116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ujnvwid/_remote_module_non_scriptable.py 2022-09-27T16:02:52.6253781Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:02:52.6263761Z 2022-09-27T16:02:52.6263875Z Running tests... 2022-09-27T16:02:52.6264430Z ---------------------------------------------------------------------- 2022-09-27T16:02:52.8728674Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 683 2022-09-27T16:02:52.8754473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 684 2022-09-27T16:02:52.8782063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 685 2022-09-27T16:02:52.8810349Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 686 2022-09-27T16:02:53.9541153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:53.9541726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:53.9542495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:53.9543052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:53.9716071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:53.9716765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:53.9914143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:53.9914944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:53.9916652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:53.9917240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:54.0079436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:54.0080101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:54.0238585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:54.0239205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:54.0599935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:54.0600725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:54.1179550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hgdbv19 2022-09-27T16:02:54.1180194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpslii72me 2022-09-27T16:02:54.1180844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hgdbv19/_remote_module_non_scriptable.py 2022-09-27T16:02:54.1181532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpslii72me/_remote_module_non_scriptable.py 2022-09-27T16:02:54.1334937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgy7t5s8t 2022-09-27T16:02:54.1336362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgy7t5s8t/_remote_module_non_scriptable.py 2022-09-27T16:02:54.1867977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_ulas6v 2022-09-27T16:02:54.1868700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_ulas6v/_remote_module_non_scriptable.py 2022-09-27T16:02:54.3756658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:02:54.3905144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:02:54.3965985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:02:54.4774188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:02:54.7864202Z skip: Need at least 1 CUDA device (2.160s) 2022-09-27T16:02:54.7864391Z 2022-09-27T16:02:54.7864689Z ---------------------------------------------------------------------- 2022-09-27T16:02:54.7864941Z Ran 1 test in 2.160s 2022-09-27T16:02:54.7865060Z 2022-09-27T16:02:54.7865148Z OK (skipped=1) 2022-09-27T16:02:54.7865258Z 2022-09-27T16:02:54.7865331Z Generating XML reports... 2022-09-27T16:02:54.7911653Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220927160252.xml 2022-09-27T16:02:56.1459410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:56.1459798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:56.1466881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:56.1467418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:56.3038432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfl7xvt7l 2022-09-27T16:02:56.3039513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfl7xvt7l/_remote_module_non_scriptable.py 2022-09-27T16:02:56.5618412Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:02:56.5628902Z 2022-09-27T16:02:56.5629078Z Running tests... 2022-09-27T16:02:56.5629571Z ---------------------------------------------------------------------- 2022-09-27T16:02:56.8022233Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 738 2022-09-27T16:02:56.8048581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 739 2022-09-27T16:02:56.8075966Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 740 2022-09-27T16:02:56.8103703Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 741 2022-09-27T16:02:57.9300183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:57.9384418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:57.9385392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:57.9386129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:57.9459395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:57.9459888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:57.9676055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:57.9676686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:57.9746918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:02:57.9747583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:02:57.9761583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:57.9762253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:57.9837831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:57.9838454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:58.0101584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:02:58.0102268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:02:58.0945221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpimmy9552 2022-09-27T16:02:58.0946573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpimmy9552/_remote_module_non_scriptable.py 2022-09-27T16:02:58.1022832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7cqvl_d 2022-09-27T16:02:58.1023684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7cqvl_d/_remote_module_non_scriptable.py 2022-09-27T16:02:58.1110112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7x96lpft 2022-09-27T16:02:58.1111281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7x96lpft/_remote_module_non_scriptable.py 2022-09-27T16:02:58.1356904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphishxprq 2022-09-27T16:02:58.1357999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphishxprq/_remote_module_non_scriptable.py 2022-09-27T16:02:58.3519046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:02:58.3582572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:02:58.3692527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:02:58.3915352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:02:58.7157303Z skip: Need at least 1 CUDA device (2.152s) 2022-09-27T16:02:58.7157612Z 2022-09-27T16:02:58.7158125Z ---------------------------------------------------------------------- 2022-09-27T16:02:58.7158483Z Ran 1 test in 2.153s 2022-09-27T16:02:58.7158598Z 2022-09-27T16:02:58.7158670Z OK (skipped=1) 2022-09-27T16:02:58.7158777Z 2022-09-27T16:02:58.7158861Z Generating XML reports... 2022-09-27T16:02:58.7204896Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220927160256.xml 2022-09-27T16:03:00.0809420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:00.0809792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:00.0817737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:00.0818100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:00.2381349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7r9tryy_ 2022-09-27T16:03:00.2383185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7r9tryy_/_remote_module_non_scriptable.py 2022-09-27T16:03:00.4959381Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:00.4969009Z 2022-09-27T16:03:00.4969094Z Running tests... 2022-09-27T16:03:00.4969633Z ---------------------------------------------------------------------- 2022-09-27T16:03:00.7399559Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 793 2022-09-27T16:03:00.7425771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 794 2022-09-27T16:03:00.7452920Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 795 2022-09-27T16:03:00.7482795Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 796 2022-09-27T16:03:01.8463626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:01.8464298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:01.8656419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:01.8656943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:01.8733688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:01.8734285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:01.8852664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:01.8853213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:01.8858949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:01.8859478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:01.9062597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:01.9063222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:01.9138934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:01.9139579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:01.9254110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:01.9254721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:02.0148658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqi6w7p3l 2022-09-27T16:03:02.0150429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqi6w7p3l/_remote_module_non_scriptable.py 2022-09-27T16:03:02.0347373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbnxmj5e 2022-09-27T16:03:02.0348306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbnxmj5e/_remote_module_non_scriptable.py 2022-09-27T16:03:02.0430417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5km89ir 2022-09-27T16:03:02.0431421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5km89ir/_remote_module_non_scriptable.py 2022-09-27T16:03:02.0535069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcq5dcv_ 2022-09-27T16:03:02.0535919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcq5dcv_/_remote_module_non_scriptable.py 2022-09-27T16:03:02.2716970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:02.2888664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:02.3000393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:03:02.3079655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:03:02.6535389Z skip: Need at least 1 CUDA device (2.156s) 2022-09-27T16:03:02.6535690Z 2022-09-27T16:03:02.6536154Z ---------------------------------------------------------------------- 2022-09-27T16:03:02.6536419Z Ran 1 test in 2.157s 2022-09-27T16:03:02.6536534Z 2022-09-27T16:03:02.6536608Z OK (skipped=1) 2022-09-27T16:03:02.6536718Z 2022-09-27T16:03:02.6536805Z Generating XML reports... 2022-09-27T16:03:02.6580589Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220927160300.xml 2022-09-27T16:03:04.0238452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:04.0238825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:04.0246551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:04.0246911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:04.1810656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphyni7g1r 2022-09-27T16:03:04.1811552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphyni7g1r/_remote_module_non_scriptable.py 2022-09-27T16:03:04.4378298Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:04.4389370Z 2022-09-27T16:03:04.4389966Z Running tests... 2022-09-27T16:03:04.4390342Z ---------------------------------------------------------------------- 2022-09-27T16:03:04.6843991Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 848 2022-09-27T16:03:04.6871132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 849 2022-09-27T16:03:05.7081954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:05.7082341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:05.7379804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:05.7437316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:05.7437829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:05.7438181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:05.7728896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:05.7729515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:05.8683982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpold53n7x 2022-09-27T16:03:05.8684890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpold53n7x/_remote_module_non_scriptable.py 2022-09-27T16:03:05.8975871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf11kgmo5 2022-09-27T16:03:05.8977444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf11kgmo5/_remote_module_non_scriptable.py 2022-09-27T16:03:06.1201551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:06.1530108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:06.3912747Z skip: Need at least 1 CUDA device (1.952s) 2022-09-27T16:03:06.3913053Z 2022-09-27T16:03:06.3913556Z ---------------------------------------------------------------------- 2022-09-27T16:03:06.3913874Z Ran 1 test in 1.952s 2022-09-27T16:03:06.3913978Z 2022-09-27T16:03:06.3914052Z OK (skipped=1) 2022-09-27T16:03:06.3914163Z 2022-09-27T16:03:06.3914250Z Generating XML reports... 2022-09-27T16:03:06.3958598Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220927160304.xml 2022-09-27T16:03:07.7568915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:07.7569333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:07.7577207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:07.7577601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:07.9166986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75p_agn8 2022-09-27T16:03:07.9167605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75p_agn8/_remote_module_non_scriptable.py 2022-09-27T16:03:08.1767682Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:08.1778703Z 2022-09-27T16:03:08.1779129Z Running tests... 2022-09-27T16:03:08.1779734Z ---------------------------------------------------------------------- 2022-09-27T16:03:08.4240829Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 881 2022-09-27T16:03:08.4268715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 882 2022-09-27T16:03:09.4800023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:09.4800462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:09.4912364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:09.4912731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:09.5146189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:09.5146664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:09.5255865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:09.5256380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:09.6389790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzsjyp2c 2022-09-27T16:03:09.6390348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzsjyp2c/_remote_module_non_scriptable.py 2022-09-27T16:03:09.6498377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5wznoju 2022-09-27T16:03:09.6499664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5wznoju/_remote_module_non_scriptable.py 2022-09-27T16:03:09.8946011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:09.9060554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:10.2311174Z skip: Need at least 1 CUDA device (2.053s) 2022-09-27T16:03:10.2311492Z 2022-09-27T16:03:10.2311942Z ---------------------------------------------------------------------- 2022-09-27T16:03:10.2312197Z Ran 1 test in 2.053s 2022-09-27T16:03:10.2312533Z 2022-09-27T16:03:10.2312609Z OK (skipped=1) 2022-09-27T16:03:10.2312707Z 2022-09-27T16:03:10.2312796Z Generating XML reports... 2022-09-27T16:03:10.2356660Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220927160308.xml 2022-09-27T16:03:11.6095636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:11.6096006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:11.6104633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:11.6105001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:11.7716559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5uye4g9t 2022-09-27T16:03:11.7717641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5uye4g9t/_remote_module_non_scriptable.py 2022-09-27T16:03:12.0310452Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:12.0321440Z 2022-09-27T16:03:12.0321776Z Running tests... 2022-09-27T16:03:12.0322392Z ---------------------------------------------------------------------- 2022-09-27T16:03:12.2798302Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 914 2022-09-27T16:03:12.2824914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 915 2022-09-27T16:03:13.3200336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:13.3200705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:13.3300102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:13.3300461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:13.3549880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:13.3550251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:13.3656672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:13.3657031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:13.4801148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpina1szeu 2022-09-27T16:03:13.4802483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpina1szeu/_remote_module_non_scriptable.py 2022-09-27T16:03:13.4900736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5xo2x3u 2022-09-27T16:03:13.4902578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5xo2x3u/_remote_module_non_scriptable.py 2022-09-27T16:03:13.7341397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:13.7440545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:13.9866837Z skip: Need at least 1 CUDA device (1.954s) 2022-09-27T16:03:13.9867062Z 2022-09-27T16:03:13.9867444Z ---------------------------------------------------------------------- 2022-09-27T16:03:13.9867699Z Ran 1 test in 1.954s 2022-09-27T16:03:13.9867815Z 2022-09-27T16:03:13.9867876Z OK (skipped=1) 2022-09-27T16:03:13.9867985Z 2022-09-27T16:03:13.9868072Z Generating XML reports... 2022-09-27T16:03:13.9912601Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220927160312.xml 2022-09-27T16:03:15.3407837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:15.3408463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:15.3416180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:15.3416776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:15.5038675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp384fjhl8 2022-09-27T16:03:15.5039462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp384fjhl8/_remote_module_non_scriptable.py 2022-09-27T16:03:15.7605563Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:15.7616615Z 2022-09-27T16:03:15.7616719Z Running tests... 2022-09-27T16:03:15.7617274Z ---------------------------------------------------------------------- 2022-09-27T16:03:16.0094782Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 947 2022-09-27T16:03:16.0121910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 948 2022-09-27T16:03:17.0485340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:17.0485982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:17.0503747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:17.0504370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:17.0839430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:17.0840189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:17.0859355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:17.0859926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:17.2084418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxi74zw8 2022-09-27T16:03:17.2085428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxi74zw8/_remote_module_non_scriptable.py 2022-09-27T16:03:17.2103755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrzn2mmm 2022-09-27T16:03:17.2105625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrzn2mmm/_remote_module_non_scriptable.py 2022-09-27T16:03:17.4629973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:17.4639321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:17.7162942Z skip: Need at least 1 CUDA device (1.954s) 2022-09-27T16:03:17.7163191Z 2022-09-27T16:03:17.7163720Z ---------------------------------------------------------------------- 2022-09-27T16:03:17.7164039Z Ran 1 test in 1.954s 2022-09-27T16:03:17.7164159Z 2022-09-27T16:03:17.7164242Z OK (skipped=1) 2022-09-27T16:03:17.7164355Z 2022-09-27T16:03:17.7164443Z Generating XML reports... 2022-09-27T16:03:17.7208226Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220927160315.xml 2022-09-27T16:03:19.0855596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:19.0855958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:19.0864885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:19.0865274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:19.2403557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46ct_5cd 2022-09-27T16:03:19.2404382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46ct_5cd/_remote_module_non_scriptable.py 2022-09-27T16:03:19.4975264Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:19.4985578Z 2022-09-27T16:03:19.4985693Z Running tests... 2022-09-27T16:03:19.4986137Z ---------------------------------------------------------------------- 2022-09-27T16:03:19.7437096Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 980 2022-09-27T16:03:19.7465267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 981 2022-09-27T16:03:19.7492502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 982 2022-09-27T16:03:19.7522240Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 983 2022-09-27T16:03:20.8845344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:20.8846039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:20.9087439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:20.9088032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:20.9173432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:20.9174123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:20.9184254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:20.9184898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:20.9203538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:20.9204206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:20.9467466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:20.9468144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:20.9547186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:20.9547812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:20.9557458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:20.9558206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:21.0471912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkn1bb182 2022-09-27T16:03:21.0472810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkn1bb182/_remote_module_non_scriptable.py 2022-09-27T16:03:21.0739693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkz5044m2 2022-09-27T16:03:21.0740494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkz5044m2/_remote_module_non_scriptable.py 2022-09-27T16:03:21.0844437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nfeln8e 2022-09-27T16:03:21.0845575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nfeln8e/_remote_module_non_scriptable.py 2022-09-27T16:03:21.0992379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13dhkdlz 2022-09-27T16:03:21.0993583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13dhkdlz/_remote_module_non_scriptable.py 2022-09-27T16:03:21.3024530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:03:21.3309119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:03:21.3372068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:21.3525594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:21.6575225Z skip: Need at least 2 CUDA devices (2.159s) 2022-09-27T16:03:21.6575503Z 2022-09-27T16:03:21.6575849Z ---------------------------------------------------------------------- 2022-09-27T16:03:21.6576169Z Ran 1 test in 2.159s 2022-09-27T16:03:21.6576290Z 2022-09-27T16:03:21.6576368Z OK (skipped=1) 2022-09-27T16:03:21.6576479Z 2022-09-27T16:03:21.6576565Z Generating XML reports... 2022-09-27T16:03:21.6621416Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20220927160319.xml 2022-09-27T16:03:23.0464211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:23.0464632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:23.0474841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:23.0475416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:23.2132068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoc04dhi_ 2022-09-27T16:03:23.2132860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoc04dhi_/_remote_module_non_scriptable.py 2022-09-27T16:03:23.4756857Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:23.4767339Z 2022-09-27T16:03:23.4767487Z Running tests... 2022-09-27T16:03:23.4768110Z ---------------------------------------------------------------------- 2022-09-27T16:03:23.7167096Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1035 2022-09-27T16:03:23.7195319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1036 2022-09-27T16:03:24.7449854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:24.7450212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:24.7772702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:24.7773417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:24.7805401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:24.7805941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:24.8145776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:24.8146463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:24.9061533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4x9p2tek 2022-09-27T16:03:24.9063855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4x9p2tek/_remote_module_non_scriptable.py 2022-09-27T16:03:24.9407535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpamfulgqm 2022-09-27T16:03:24.9408959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpamfulgqm/_remote_module_non_scriptable.py 2022-09-27T16:03:25.1618616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:25.2125790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:25.5238252Z skip: Need at least 4 CUDA devices (2.047s) 2022-09-27T16:03:25.5238543Z 2022-09-27T16:03:25.5239160Z ---------------------------------------------------------------------- 2022-09-27T16:03:25.5239424Z Ran 1 test in 2.047s 2022-09-27T16:03:25.5239541Z 2022-09-27T16:03:25.5239615Z OK (skipped=1) 2022-09-27T16:03:25.5239794Z 2022-09-27T16:03:25.5239870Z Generating XML reports... 2022-09-27T16:03:25.5283492Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160323.xml 2022-09-27T16:03:26.8878643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:26.8879299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:26.8887751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:26.8888337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:27.0449279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobw2vzxu 2022-09-27T16:03:27.0450236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobw2vzxu/_remote_module_non_scriptable.py 2022-09-27T16:03:27.3016477Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:27.3026957Z 2022-09-27T16:03:27.3027189Z Running tests... 2022-09-27T16:03:27.3027653Z ---------------------------------------------------------------------- 2022-09-27T16:03:27.5475648Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1068 2022-09-27T16:03:27.5501797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1069 2022-09-27T16:03:28.5877695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:28.5878303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:28.6208728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:28.6209379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:28.6316077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:28.6316436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:28.6671144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:28.6671735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:28.7456426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9d5mk_qf 2022-09-27T16:03:28.7457491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9d5mk_qf/_remote_module_non_scriptable.py 2022-09-27T16:03:28.7936184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2himcpc 2022-09-27T16:03:28.7937940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2himcpc/_remote_module_non_scriptable.py 2022-09-27T16:03:28.9991163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:29.0493901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:29.3548733Z skip: Need at least 4 CUDA devices (2.052s) 2022-09-27T16:03:29.3548996Z 2022-09-27T16:03:29.3549288Z ---------------------------------------------------------------------- 2022-09-27T16:03:29.3549660Z Ran 1 test in 2.052s 2022-09-27T16:03:29.3549840Z 2022-09-27T16:03:29.3549917Z OK (skipped=1) 2022-09-27T16:03:29.3550082Z 2022-09-27T16:03:29.3550212Z Generating XML reports... 2022-09-27T16:03:29.3595252Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160327.xml 2022-09-27T16:03:30.7274065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:30.7274663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:30.7282974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:30.7283348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:30.8876773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3nxb3kft 2022-09-27T16:03:30.8877668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3nxb3kft/_remote_module_non_scriptable.py 2022-09-27T16:03:31.1446775Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:31.1457679Z 2022-09-27T16:03:31.1458045Z Running tests... 2022-09-27T16:03:31.1458442Z ---------------------------------------------------------------------- 2022-09-27T16:03:31.3918108Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1101 2022-09-27T16:03:31.3945803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1102 2022-09-27T16:03:32.4245714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:32.4246108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:32.4590230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:32.4590609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:32.4627666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:32.4628017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:32.4976338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:32.4976713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:32.5842113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1k366ruc 2022-09-27T16:03:32.5842816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1k366ruc/_remote_module_non_scriptable.py 2022-09-27T16:03:32.6225726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkffcv9ry 2022-09-27T16:03:32.6226991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkffcv9ry/_remote_module_non_scriptable.py 2022-09-27T16:03:32.8411701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:32.8804867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:33.1992923Z skip: Need at least 4 CUDA devices (2.053s) 2022-09-27T16:03:33.1993191Z 2022-09-27T16:03:33.1993546Z ---------------------------------------------------------------------- 2022-09-27T16:03:33.1993811Z Ran 1 test in 2.053s 2022-09-27T16:03:33.1993981Z 2022-09-27T16:03:33.1994060Z OK (skipped=1) 2022-09-27T16:03:33.1994175Z 2022-09-27T16:03:33.1994261Z Generating XML reports... 2022-09-27T16:03:33.2040725Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160331.xml 2022-09-27T16:03:34.5691648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:34.5692025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:34.5700307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:34.5700674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:34.7291818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_oa1cbo 2022-09-27T16:03:34.7292589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_oa1cbo/_remote_module_non_scriptable.py 2022-09-27T16:03:34.9867182Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:34.9877398Z 2022-09-27T16:03:34.9877631Z Running tests... 2022-09-27T16:03:34.9878045Z ---------------------------------------------------------------------- 2022-09-27T16:03:35.2287264Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1134 2022-09-27T16:03:35.2314304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1135 2022-09-27T16:03:36.2584057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:36.2584754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:36.2588090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:36.2588741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:36.2931881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:36.2932508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:36.2934045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:36.2934575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:36.4172835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdgmzab5 2022-09-27T16:03:36.4173473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtdgfxcr 2022-09-27T16:03:36.4174133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdgmzab5/_remote_module_non_scriptable.py 2022-09-27T16:03:36.4174826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtdgfxcr/_remote_module_non_scriptable.py 2022-09-27T16:03:36.6731493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:36.6744208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:36.9354132Z skip: Need at least 4 CUDA devices (1.947s) 2022-09-27T16:03:36.9354349Z 2022-09-27T16:03:36.9354852Z ---------------------------------------------------------------------- 2022-09-27T16:03:36.9355107Z Ran 1 test in 1.948s 2022-09-27T16:03:36.9355223Z 2022-09-27T16:03:36.9355298Z OK (skipped=1) 2022-09-27T16:03:36.9355394Z 2022-09-27T16:03:36.9355480Z Generating XML reports... 2022-09-27T16:03:36.9402214Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160334.xml 2022-09-27T16:03:38.2993735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:38.2994197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:38.3002135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:38.3002483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:38.4569139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3uxd13om 2022-09-27T16:03:38.4570107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3uxd13om/_remote_module_non_scriptable.py 2022-09-27T16:03:38.7144176Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:38.7154339Z 2022-09-27T16:03:38.7154824Z Running tests... 2022-09-27T16:03:38.7155515Z ---------------------------------------------------------------------- 2022-09-27T16:03:38.7159725Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:03:38.7160348Z 2022-09-27T16:03:38.7160660Z ---------------------------------------------------------------------- 2022-09-27T16:03:38.7161102Z Ran 1 test in 0.001s 2022-09-27T16:03:38.7161311Z 2022-09-27T16:03:38.7161427Z OK (skipped=1) 2022-09-27T16:03:38.7161555Z 2022-09-27T16:03:38.7161642Z Generating XML reports... 2022-09-27T16:03:38.7191245Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160338.xml 2022-09-27T16:03:39.9477275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:39.9477942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:39.9486796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:39.9487437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:40.1030579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmk1loxp 2022-09-27T16:03:40.1031715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmk1loxp/_remote_module_non_scriptable.py 2022-09-27T16:03:40.3625017Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:40.3636063Z 2022-09-27T16:03:40.3636393Z Running tests... 2022-09-27T16:03:40.3637014Z ---------------------------------------------------------------------- 2022-09-27T16:03:40.3641821Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:03:40.3642343Z 2022-09-27T16:03:40.3642713Z ---------------------------------------------------------------------- 2022-09-27T16:03:40.3643152Z Ran 1 test in 0.001s 2022-09-27T16:03:40.3643362Z 2022-09-27T16:03:40.3643495Z OK (skipped=1) 2022-09-27T16:03:40.3643608Z 2022-09-27T16:03:40.3643697Z Generating XML reports... 2022-09-27T16:03:40.3672785Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160340.xml 2022-09-27T16:03:41.6146472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:41.6146919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:41.6155134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:41.6155521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:41.7728171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3n1yx70n 2022-09-27T16:03:41.7729553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3n1yx70n/_remote_module_non_scriptable.py 2022-09-27T16:03:42.0358203Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:42.0368308Z 2022-09-27T16:03:42.0368427Z Running tests... 2022-09-27T16:03:42.0368864Z ---------------------------------------------------------------------- 2022-09-27T16:03:42.0373718Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:03:42.0374112Z 2022-09-27T16:03:42.0374573Z ---------------------------------------------------------------------- 2022-09-27T16:03:42.0374857Z Ran 1 test in 0.001s 2022-09-27T16:03:42.0374963Z 2022-09-27T16:03:42.0375039Z OK (skipped=1) 2022-09-27T16:03:42.0375154Z 2022-09-27T16:03:42.0375240Z Generating XML reports... 2022-09-27T16:03:42.0404500Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160342.xml 2022-09-27T16:03:43.2800378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:43.2801022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:43.2808676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:43.2809039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:43.4376281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcqqcimt1 2022-09-27T16:03:43.4377422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcqqcimt1/_remote_module_non_scriptable.py 2022-09-27T16:03:43.6949445Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:43.6960887Z 2022-09-27T16:03:43.6961301Z Running tests... 2022-09-27T16:03:43.6961791Z ---------------------------------------------------------------------- 2022-09-27T16:03:43.6966121Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:03:43.6967013Z 2022-09-27T16:03:43.6967287Z ---------------------------------------------------------------------- 2022-09-27T16:03:43.6967695Z Ran 1 test in 0.001s 2022-09-27T16:03:43.6967887Z 2022-09-27T16:03:43.6968015Z OK (skipped=1) 2022-09-27T16:03:43.6968210Z 2022-09-27T16:03:43.6968326Z Generating XML reports... 2022-09-27T16:03:43.6999025Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160343.xml 2022-09-27T16:03:44.9408206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:44.9408602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:44.9416569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:44.9416997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:45.0991586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxsu4fsqq 2022-09-27T16:03:45.0992326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxsu4fsqq/_remote_module_non_scriptable.py 2022-09-27T16:03:45.3564978Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:45.3574126Z 2022-09-27T16:03:45.3574251Z Running tests... 2022-09-27T16:03:45.3574824Z ---------------------------------------------------------------------- 2022-09-27T16:03:45.5999092Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1207 2022-09-27T16:03:45.6025972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1208 2022-09-27T16:03:45.6052948Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1209 2022-09-27T16:03:45.6081309Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1210 2022-09-27T16:03:46.6967186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:46.6967849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:46.6983611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:46.6984294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:46.7320724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:46.7321871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:46.7334313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:46.7335233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:46.7350076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:46.7350630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:46.7552435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:46.7553059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:46.7685414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:46.7686048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:46.7921473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:46.7922155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:46.8607509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52k5gfqz 2022-09-27T16:03:46.8608298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52k5gfqz/_remote_module_non_scriptable.py 2022-09-27T16:03:46.8611250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplf_hav7i 2022-09-27T16:03:46.8613732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplf_hav7i/_remote_module_non_scriptable.py 2022-09-27T16:03:46.8946437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr8og24d5 2022-09-27T16:03:46.8948055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr8og24d5/_remote_module_non_scriptable.py 2022-09-27T16:03:46.9188913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5z9wopmv 2022-09-27T16:03:46.9190126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5z9wopmv/_remote_module_non_scriptable.py 2022-09-27T16:03:47.1164521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:03:47.1199486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:03:47.1509343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:47.1887666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:47.5134308Z skip: Need at least 1 CUDA device (2.156s) 2022-09-27T16:03:47.5134619Z 2022-09-27T16:03:47.5135120Z ---------------------------------------------------------------------- 2022-09-27T16:03:47.5135406Z Ran 1 test in 2.156s 2022-09-27T16:03:47.5135520Z 2022-09-27T16:03:47.5135592Z OK (skipped=1) 2022-09-27T16:03:47.5135713Z 2022-09-27T16:03:47.5135798Z Generating XML reports... 2022-09-27T16:03:47.5180071Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160345.xml 2022-09-27T16:03:48.8981362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:48.8981740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:48.8989224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:48.8989565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:49.0559787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpecqxel0t 2022-09-27T16:03:49.0560747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpecqxel0t/_remote_module_non_scriptable.py 2022-09-27T16:03:49.3121729Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:49.3131460Z 2022-09-27T16:03:49.3131581Z Running tests... 2022-09-27T16:03:49.3132008Z ---------------------------------------------------------------------- 2022-09-27T16:03:49.5487579Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1262 2022-09-27T16:03:49.5515154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1263 2022-09-27T16:03:49.5542297Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1264 2022-09-27T16:03:49.5571004Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1265 2022-09-27T16:03:50.6692200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:50.6692915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:50.6795601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:50.6796252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:50.7060860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:50.7061524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:50.7158651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:50.7159230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:50.7242262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:50.7242913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:50.7532467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:50.7533113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:50.7603357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:50.7604013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:50.7903472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:50.7904109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:50.8346720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzwb1y8z3 2022-09-27T16:03:50.8347958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzwb1y8z3/_remote_module_non_scriptable.py 2022-09-27T16:03:50.8457461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbp74srzp 2022-09-27T16:03:50.8458244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbp74srzp/_remote_module_non_scriptable.py 2022-09-27T16:03:50.8887785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcdtxsy8e 2022-09-27T16:03:50.8888595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcdtxsy8e/_remote_module_non_scriptable.py 2022-09-27T16:03:50.9321046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdtaurj_ 2022-09-27T16:03:50.9322239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdtaurj_/_remote_module_non_scriptable.py 2022-09-27T16:03:51.0937639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:51.0998907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:51.1533088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:03:51.1899570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:03:51.4625370Z skip: Need at least 1 CUDA device (2.149s) 2022-09-27T16:03:51.4625659Z 2022-09-27T16:03:51.4626180Z ---------------------------------------------------------------------- 2022-09-27T16:03:51.4626468Z Ran 1 test in 2.149s 2022-09-27T16:03:51.4626584Z 2022-09-27T16:03:51.4626644Z OK (skipped=1) 2022-09-27T16:03:51.4626753Z 2022-09-27T16:03:51.4626838Z Generating XML reports... 2022-09-27T16:03:51.4670736Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160349.xml 2022-09-27T16:03:52.8506336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:52.8506776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:52.8515298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:52.8515651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:53.0112572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmc2r3bb6 2022-09-27T16:03:53.0113418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmc2r3bb6/_remote_module_non_scriptable.py 2022-09-27T16:03:53.2714292Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:53.2724445Z 2022-09-27T16:03:53.2724540Z Running tests... 2022-09-27T16:03:53.2725761Z ---------------------------------------------------------------------- 2022-09-27T16:03:53.5210885Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1317 2022-09-27T16:03:53.5236852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1318 2022-09-27T16:03:53.5264789Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1319 2022-09-27T16:03:53.5294213Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1320 2022-09-27T16:03:54.5967188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:54.5967835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:54.6309868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:54.6310514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:54.6364599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:54.6365167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:54.6458967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:54.6459589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:54.6489412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:54.6490028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:54.6730476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:54.6731003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:54.6837538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:54.6838256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:54.6865638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:54.6866247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:54.7574261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0i5o3s8f 2022-09-27T16:03:54.7575063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0i5o3s8f/_remote_module_non_scriptable.py 2022-09-27T16:03:54.7983785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3m8h9x1 2022-09-27T16:03:54.7984984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3m8h9x1/_remote_module_non_scriptable.py 2022-09-27T16:03:54.8096980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2yy48zhm 2022-09-27T16:03:54.8098525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2yy48zhm/_remote_module_non_scriptable.py 2022-09-27T16:03:54.8112881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzieeo9kr 2022-09-27T16:03:54.8114880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzieeo9kr/_remote_module_non_scriptable.py 2022-09-27T16:03:55.0315308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:55.0529318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:03:55.0680728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:03:55.0696284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:55.4346275Z skip: Need at least 2 CUDA devices (2.162s) 2022-09-27T16:03:55.4346475Z 2022-09-27T16:03:55.4346997Z ---------------------------------------------------------------------- 2022-09-27T16:03:55.4347377Z Ran 1 test in 2.162s 2022-09-27T16:03:55.4347489Z 2022-09-27T16:03:55.4347561Z OK (skipped=1) 2022-09-27T16:03:55.4347669Z 2022-09-27T16:03:55.4347751Z Generating XML reports... 2022-09-27T16:03:55.4392152Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160353.xml 2022-09-27T16:03:56.8077628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:56.8077988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:56.8086137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:56.8086497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:56.9659648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpli96wlcu 2022-09-27T16:03:56.9660299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpli96wlcu/_remote_module_non_scriptable.py 2022-09-27T16:03:57.2241574Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:03:57.2252311Z 2022-09-27T16:03:57.2252637Z Running tests... 2022-09-27T16:03:57.2253280Z ---------------------------------------------------------------------- 2022-09-27T16:03:57.4649367Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1372 2022-09-27T16:03:57.4676548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1373 2022-09-27T16:03:57.4703735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1374 2022-09-27T16:03:57.4731840Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1375 2022-09-27T16:03:58.6117608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:58.6118235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:58.6429696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:58.6430415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:58.6460374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:58.6461006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:58.6650650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:58.6651278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:58.6736451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:03:58.6737066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:03:58.6800191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:58.6800817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:58.7033869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:58.7034710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:58.7119519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:03:58.7120180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:03:58.7737145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjsnm20m 2022-09-27T16:03:58.7739103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjsnm20m/_remote_module_non_scriptable.py 2022-09-27T16:03:58.8072697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9s86rte 2022-09-27T16:03:58.8073848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9s86rte/_remote_module_non_scriptable.py 2022-09-27T16:03:58.8295979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3ll4uh7 2022-09-27T16:03:58.8297576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3ll4uh7/_remote_module_non_scriptable.py 2022-09-27T16:03:58.8384782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaf9p82oz 2022-09-27T16:03:58.8386195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaf9p82oz/_remote_module_non_scriptable.py 2022-09-27T16:03:59.0304287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:03:59.0658987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:03:59.0873262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:03:59.0943552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:03:59.3786505Z skip: Need at least 1 CUDA device (2.153s) 2022-09-27T16:03:59.3786851Z 2022-09-27T16:03:59.3787229Z ---------------------------------------------------------------------- 2022-09-27T16:03:59.3787468Z Ran 1 test in 2.153s 2022-09-27T16:03:59.3787585Z 2022-09-27T16:03:59.3787658Z OK (skipped=1) 2022-09-27T16:03:59.3787767Z 2022-09-27T16:03:59.3787857Z Generating XML reports... 2022-09-27T16:03:59.3830742Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160357.xml 2022-09-27T16:04:00.7544114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:00.7544493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:00.7553284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:00.7553654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:00.9161986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbjuby_3 2022-09-27T16:04:00.9163531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbjuby_3/_remote_module_non_scriptable.py 2022-09-27T16:04:01.1735442Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:01.1745443Z 2022-09-27T16:04:01.1745627Z Running tests... 2022-09-27T16:04:01.1746025Z ---------------------------------------------------------------------- 2022-09-27T16:04:01.4185750Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1427 2022-09-27T16:04:01.4213155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1428 2022-09-27T16:04:01.4240889Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1429 2022-09-27T16:04:01.4269509Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1430 2022-09-27T16:04:02.5388620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:02.5389205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:02.5488975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:02.5489605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:02.5501153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:02.5501762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:02.5738097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:02.5738606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:02.5761526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:02.5762209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:02.5864223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:02.5864753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:02.5882325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:02.5882824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:02.6108925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:02.6109590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:02.7030979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9l1hdoi 2022-09-27T16:04:02.7031737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9l1hdoi/_remote_module_non_scriptable.py 2022-09-27T16:04:02.7132140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnamw3_8x 2022-09-27T16:04:02.7133091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnamw3_8x/_remote_module_non_scriptable.py 2022-09-27T16:04:02.7153522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzthx91va 2022-09-27T16:04:02.7155495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzthx91va/_remote_module_non_scriptable.py 2022-09-27T16:04:02.7377893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpembd1_tp 2022-09-27T16:04:02.7378894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpembd1_tp/_remote_module_non_scriptable.py 2022-09-27T16:04:02.9591942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:02.9689989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:02.9713407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:02.9965608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:03.3324808Z skip: Need at least 1 CUDA device (2.158s) 2022-09-27T16:04:03.3325095Z 2022-09-27T16:04:03.3325402Z ---------------------------------------------------------------------- 2022-09-27T16:04:03.3326246Z Ran 1 test in 2.158s 2022-09-27T16:04:03.3326454Z 2022-09-27T16:04:03.3326572Z OK (skipped=1) 2022-09-27T16:04:03.3326755Z 2022-09-27T16:04:03.3326916Z Generating XML reports... 2022-09-27T16:04:03.3370943Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160401.xml 2022-09-27T16:04:04.7203264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:04.7203649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:04.7211740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:04.7212260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:04.8776402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphyw8bbzp 2022-09-27T16:04:04.8777449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphyw8bbzp/_remote_module_non_scriptable.py 2022-09-27T16:04:05.1353242Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:05.1363747Z 2022-09-27T16:04:05.3792095Z Running tests... 2022-09-27T16:04:05.3792605Z ---------------------------------------------------------------------- 2022-09-27T16:04:05.3793126Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1482 2022-09-27T16:04:05.3819636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1483 2022-09-27T16:04:05.3847285Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1484 2022-09-27T16:04:05.3876989Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1485 2022-09-27T16:04:06.5001983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:06.5002628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:06.5167658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:06.5168276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:06.5244391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:06.5244957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:06.5379767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:06.5380376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:06.5548059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:06.5548789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:06.5555716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:06.5556345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:06.5620961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:06.5621503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:06.5931119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:06.5931770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:06.6659433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvx15_biy 2022-09-27T16:04:06.6660144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvx15_biy/_remote_module_non_scriptable.py 2022-09-27T16:04:06.6815239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnj507mb3 2022-09-27T16:04:06.6816263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnj507mb3/_remote_module_non_scriptable.py 2022-09-27T16:04:06.6896532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphs46xqr2 2022-09-27T16:04:06.6897949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphs46xqr2/_remote_module_non_scriptable.py 2022-09-27T16:04:06.7192986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxwpmq0e 2022-09-27T16:04:06.7193809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxwpmq0e/_remote_module_non_scriptable.py 2022-09-27T16:04:06.9244446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:06.9382025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:06.9494610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:06.9881716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:07.2930519Z skip: Need at least 1 CUDA device (2.156s) 2022-09-27T16:04:07.2930798Z 2022-09-27T16:04:07.2931269Z ---------------------------------------------------------------------- 2022-09-27T16:04:07.2931698Z Ran 1 test in 2.157s 2022-09-27T16:04:07.2931903Z 2022-09-27T16:04:07.2932027Z OK (skipped=1) 2022-09-27T16:04:07.2932217Z 2022-09-27T16:04:07.2932376Z Generating XML reports... 2022-09-27T16:04:07.2977830Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160405.xml 2022-09-27T16:04:08.6985983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:08.6986580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:08.6995215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:08.6995846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:08.8586499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0ufm0nn 2022-09-27T16:04:08.8587383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0ufm0nn/_remote_module_non_scriptable.py 2022-09-27T16:04:09.1171870Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:09.1181991Z 2022-09-27T16:04:09.1182566Z Running tests... 2022-09-27T16:04:09.1183094Z ---------------------------------------------------------------------- 2022-09-27T16:04:09.3619128Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1537 2022-09-27T16:04:09.3647579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1538 2022-09-27T16:04:09.3674464Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1539 2022-09-27T16:04:09.3703310Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1540 2022-09-27T16:04:10.4774168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:10.4774853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:10.4939107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:10.4939715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:10.5004376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:10.5005110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:10.5005945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:10.5006561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:10.5167569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:10.5168081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:10.5337368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:10.5338101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:10.5403331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:10.5403973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:10.5404815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:10.5405409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:10.6443548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4q0evnhg 2022-09-27T16:04:10.6612376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4q0evnhg/_remote_module_non_scriptable.py 2022-09-27T16:04:10.6613150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdbvmgg38 2022-09-27T16:04:10.6613882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdbvmgg38/_remote_module_non_scriptable.py 2022-09-27T16:04:10.6678138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdcti3zpi 2022-09-27T16:04:10.6679347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdcti3zpi/_remote_module_non_scriptable.py 2022-09-27T16:04:10.6681289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpej_1kd08 2022-09-27T16:04:10.6683694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpej_1kd08/_remote_module_non_scriptable.py 2022-09-27T16:04:10.9004091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:10.9190580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:10.9258911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:10.9318606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:11.2756141Z skip: Need at least 1 CUDA device (2.157s) 2022-09-27T16:04:11.2756318Z 2022-09-27T16:04:11.2756623Z ---------------------------------------------------------------------- 2022-09-27T16:04:11.2756934Z Ran 1 test in 2.157s 2022-09-27T16:04:11.2757045Z 2022-09-27T16:04:11.2757118Z OK (skipped=1) 2022-09-27T16:04:11.2757225Z 2022-09-27T16:04:11.2757310Z Generating XML reports... 2022-09-27T16:04:11.2800530Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160409.xml 2022-09-27T16:04:12.6514333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:12.6514826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:12.6522794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:12.6523336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:12.8113242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp951ew3gq 2022-09-27T16:04:12.8114304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp951ew3gq/_remote_module_non_scriptable.py 2022-09-27T16:04:13.0664316Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:13.0674472Z 2022-09-27T16:04:13.0674751Z Running tests... 2022-09-27T16:04:13.0675403Z ---------------------------------------------------------------------- 2022-09-27T16:04:13.3116130Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1592 2022-09-27T16:04:13.3143957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1593 2022-09-27T16:04:13.3171344Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1594 2022-09-27T16:04:13.3200357Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1595 2022-09-27T16:04:14.4485138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:14.4485804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:14.4829182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:14.4829815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:14.4834970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:14.4835588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:14.4852288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:14.4852953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:14.5027633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:14.5028124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:14.5220285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:14.5220942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:14.5230276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:14.5230986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:14.5403775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:14.5404409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:14.6134372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaeqkgerp 2022-09-27T16:04:14.6135161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaeqkgerp/_remote_module_non_scriptable.py 2022-09-27T16:04:14.6504965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5au0os77 2022-09-27T16:04:14.6506814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5au0os77/_remote_module_non_scriptable.py 2022-09-27T16:04:14.6674308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7e0hu4t 2022-09-27T16:04:14.6675553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7e0hu4t/_remote_module_non_scriptable.py 2022-09-27T16:04:14.6683538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnzzhr0u 2022-09-27T16:04:14.6685668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnzzhr0u/_remote_module_non_scriptable.py 2022-09-27T16:04:14.8678924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:14.9051495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:14.9202708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:14.9401709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:15.2255008Z skip: Need at least 1 CUDA device (2.158s) 2022-09-27T16:04:15.2255206Z 2022-09-27T16:04:15.2255501Z ---------------------------------------------------------------------- 2022-09-27T16:04:15.2255762Z Ran 1 test in 2.158s 2022-09-27T16:04:15.2255882Z 2022-09-27T16:04:15.2255954Z OK (skipped=1) 2022-09-27T16:04:15.2256064Z 2022-09-27T16:04:15.2256151Z Generating XML reports... 2022-09-27T16:04:15.2299225Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160413.xml 2022-09-27T16:04:16.5985201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:16.5985580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:16.5993011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:16.5993427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:16.7570796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvzhk1w3k 2022-09-27T16:04:16.7571808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvzhk1w3k/_remote_module_non_scriptable.py 2022-09-27T16:04:17.0133703Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:17.0144503Z 2022-09-27T16:04:17.0144839Z Running tests... 2022-09-27T16:04:17.0145478Z ---------------------------------------------------------------------- 2022-09-27T16:04:17.2567181Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1647 2022-09-27T16:04:17.2592606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1648 2022-09-27T16:04:17.2619876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1649 2022-09-27T16:04:17.2648294Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1650 2022-09-27T16:04:18.3624554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:18.3625157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:18.3828772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:18.3829363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:18.3849451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:18.3849965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:18.3983234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:18.3995556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:18.4104953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:18.4105598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:18.4204740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:18.4205375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:18.4218164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:18.4218743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:18.4474175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:18.4475034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:18.5242278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuuu24kll 2022-09-27T16:04:18.5243693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuuu24kll/_remote_module_non_scriptable.py 2022-09-27T16:04:18.5479845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptx263gzk 2022-09-27T16:04:18.5480524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptx263gzk/_remote_module_non_scriptable.py 2022-09-27T16:04:18.5481741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0ncsw2t 2022-09-27T16:04:18.5484785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0ncsw2t/_remote_module_non_scriptable.py 2022-09-27T16:04:18.5738536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmhokdby 2022-09-27T16:04:18.5739847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmhokdby/_remote_module_non_scriptable.py 2022-09-27T16:04:18.7834632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:18.8082765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:18.8090473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:18.8648495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:19.1702757Z skip: Need at least 1 CUDA device (2.156s) 2022-09-27T16:04:19.1703040Z 2022-09-27T16:04:19.1703586Z ---------------------------------------------------------------------- 2022-09-27T16:04:19.1704026Z Ran 1 test in 2.156s 2022-09-27T16:04:19.1704193Z 2022-09-27T16:04:19.1704270Z OK (skipped=1) 2022-09-27T16:04:19.1704382Z 2022-09-27T16:04:19.1704475Z Generating XML reports... 2022-09-27T16:04:19.1748912Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160417.xml 2022-09-27T16:04:20.5226126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:20.5226502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:20.5235323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:20.5235719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:20.6832456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9186dba0 2022-09-27T16:04:20.6833316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9186dba0/_remote_module_non_scriptable.py 2022-09-27T16:04:20.9450169Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:20.9460691Z 2022-09-27T16:04:20.9461052Z Running tests... 2022-09-27T16:04:20.9461440Z ---------------------------------------------------------------------- 2022-09-27T16:04:21.1844948Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1702 2022-09-27T16:04:21.1871619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1703 2022-09-27T16:04:21.1898769Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1704 2022-09-27T16:04:21.1927042Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1705 2022-09-27T16:04:22.2746262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:22.2746905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:22.2878917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:22.2879481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:22.3112556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:22.3113245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:22.3130285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:22.3130954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:22.3152231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:22.3152873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:22.3282279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:22.3282818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:22.3502806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:22.3503447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:22.3545237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:22.3545892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:22.4401400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpselmhexr 2022-09-27T16:04:22.4402095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpselmhexr/_remote_module_non_scriptable.py 2022-09-27T16:04:22.4548778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51iow2i6 2022-09-27T16:04:22.4549779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51iow2i6/_remote_module_non_scriptable.py 2022-09-27T16:04:22.4769950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmalr_1m 2022-09-27T16:04:22.4770766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmalr_1m/_remote_module_non_scriptable.py 2022-09-27T16:04:22.4814265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn6o5tl_o 2022-09-27T16:04:22.4815681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn6o5tl_o/_remote_module_non_scriptable.py 2022-09-27T16:04:22.6960858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:22.7127259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:22.7314022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:22.7439808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:23.0980639Z skip: Need at least 1 CUDA device (2.152s) 2022-09-27T16:04:23.0980877Z 2022-09-27T16:04:23.0981476Z ---------------------------------------------------------------------- 2022-09-27T16:04:23.0981732Z Ran 1 test in 2.152s 2022-09-27T16:04:23.0981850Z 2022-09-27T16:04:23.0981910Z OK (skipped=1) 2022-09-27T16:04:23.0982022Z 2022-09-27T16:04:23.0982111Z Generating XML reports... 2022-09-27T16:04:23.1025723Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160420.xml 2022-09-27T16:04:24.4671485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:24.4671921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:24.4679699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:24.4680074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:24.6217016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3u8oxft 2022-09-27T16:04:24.6217964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3u8oxft/_remote_module_non_scriptable.py 2022-09-27T16:04:24.8822139Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:24.8832360Z 2022-09-27T16:04:24.8832459Z Running tests... 2022-09-27T16:04:24.8832919Z ---------------------------------------------------------------------- 2022-09-27T16:04:25.1305409Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1757 2022-09-27T16:04:25.1332287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1758 2022-09-27T16:04:25.1359966Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1759 2022-09-27T16:04:25.1388266Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1760 2022-09-27T16:04:26.2828350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:26.2828910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:26.3057308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:26.3057905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:26.3059141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:26.3059758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:26.3207534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:26.3208098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:26.3449929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:26.3450569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:26.3457310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:26.3457975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:26.3458775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:26.3459364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:26.3808439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:26.3809296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:26.4475830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpya_bg28c 2022-09-27T16:04:26.4477027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpya_bg28c/_remote_module_non_scriptable.py 2022-09-27T16:04:26.4714286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9hwrgm2u 2022-09-27T16:04:26.4715212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9hwrgm2u/_remote_module_non_scriptable.py 2022-09-27T16:04:26.4724255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgd2epog 2022-09-27T16:04:26.4726235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgd2epog/_remote_module_non_scriptable.py 2022-09-27T16:04:26.5075572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizz1bwyh 2022-09-27T16:04:26.5077097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizz1bwyh/_remote_module_non_scriptable.py 2022-09-27T16:04:26.7031686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:26.7264455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:26.7292545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:26.7639426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:27.0442056Z skip: Need at least 1 CUDA device (2.161s) 2022-09-27T16:04:27.0442271Z 2022-09-27T16:04:27.0442617Z ---------------------------------------------------------------------- 2022-09-27T16:04:27.0442871Z Ran 1 test in 2.161s 2022-09-27T16:04:27.0442991Z 2022-09-27T16:04:27.0443067Z OK (skipped=1) 2022-09-27T16:04:27.0443220Z 2022-09-27T16:04:27.0443312Z Generating XML reports... 2022-09-27T16:04:27.0486776Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160424.xml 2022-09-27T16:04:28.4052387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:28.4052795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:28.4060521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:28.4060916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:28.5672661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgo8_cp4b 2022-09-27T16:04:28.5673857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgo8_cp4b/_remote_module_non_scriptable.py 2022-09-27T16:04:28.8272478Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:28.8283574Z 2022-09-27T16:04:28.8283657Z Running tests... 2022-09-27T16:04:28.8284332Z ---------------------------------------------------------------------- 2022-09-27T16:04:29.0713249Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1812 2022-09-27T16:04:29.0742367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1813 2022-09-27T16:04:29.0778645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1814 2022-09-27T16:04:29.0810714Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1815 2022-09-27T16:04:30.1343592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:30.1344201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:30.1489889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:30.1490783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:30.1704097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:30.1783198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:30.1784149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:30.1784740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:30.1797137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:30.1797805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:30.1852963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:30.1853586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:30.2149082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:30.2149829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:30.2166807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:30.2167420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:30.2972152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3_tnka_ 2022-09-27T16:04:30.2972895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3_tnka_/_remote_module_non_scriptable.py 2022-09-27T16:04:30.3119253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtyejj2k 2022-09-27T16:04:30.3120182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtyejj2k/_remote_module_non_scriptable.py 2022-09-27T16:04:30.3413631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo48wcgty 2022-09-27T16:04:30.3414350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo48wcgty/_remote_module_non_scriptable.py 2022-09-27T16:04:30.3435182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmx92cd3 2022-09-27T16:04:30.3436769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmx92cd3/_remote_module_non_scriptable.py 2022-09-27T16:04:30.5533957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:30.5674457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:30.5960589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:30.6019332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:30.8863050Z skip: Need at least 1 CUDA device (2.058s) 2022-09-27T16:04:30.8863246Z 2022-09-27T16:04:30.8863601Z ---------------------------------------------------------------------- 2022-09-27T16:04:30.8863871Z Ran 1 test in 2.058s 2022-09-27T16:04:30.8863989Z 2022-09-27T16:04:30.8864064Z OK (skipped=1) 2022-09-27T16:04:30.8864224Z 2022-09-27T16:04:30.8864315Z Generating XML reports... 2022-09-27T16:04:30.8908390Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160428.xml 2022-09-27T16:04:32.2442045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:32.2442431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:32.2450005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:32.2450518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:32.4029265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbapbi0y6 2022-09-27T16:04:32.4030152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbapbi0y6/_remote_module_non_scriptable.py 2022-09-27T16:04:32.6566558Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:32.6577068Z 2022-09-27T16:04:32.6577156Z Running tests... 2022-09-27T16:04:32.6578120Z ---------------------------------------------------------------------- 2022-09-27T16:04:32.8957534Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1867 2022-09-27T16:04:32.8983754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1868 2022-09-27T16:04:32.9011183Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1869 2022-09-27T16:04:32.9039620Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1870 2022-09-27T16:04:34.0272295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:34.0272964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:34.0349814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:34.0350412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:34.0410883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:34.0411523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:34.0649086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:34.0649620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:34.0694549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:34.0695204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:34.0730532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:34.0798686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:34.0799638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:34.0800253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:34.1044163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:34.1044813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:34.1920578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpon7a6106 2022-09-27T16:04:34.1921334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpon7a6106/_remote_module_non_scriptable.py 2022-09-27T16:04:34.2001870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfaow9qnf 2022-09-27T16:04:34.2003361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfaow9qnf/_remote_module_non_scriptable.py 2022-09-27T16:04:34.2071404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyuahpva4 2022-09-27T16:04:34.2073500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyuahpva4/_remote_module_non_scriptable.py 2022-09-27T16:04:34.2313367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80eh1sda 2022-09-27T16:04:34.2314124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80eh1sda/_remote_module_non_scriptable.py 2022-09-27T16:04:34.4491546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:34.4569025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:34.4859582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:34.4903810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:34.8093983Z skip: Need at least 1 CUDA device (2.151s) 2022-09-27T16:04:34.8094205Z 2022-09-27T16:04:34.8094515Z ---------------------------------------------------------------------- 2022-09-27T16:04:34.8094871Z Ran 1 test in 2.152s 2022-09-27T16:04:34.8095079Z 2022-09-27T16:04:34.8095195Z OK (skipped=1) 2022-09-27T16:04:34.8095400Z 2022-09-27T16:04:34.8095561Z Generating XML reports... 2022-09-27T16:04:34.8139324Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160432.xml 2022-09-27T16:04:36.1719738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:36.1720124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:36.1727752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:36.1728139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:36.3284726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjw3tcpz 2022-09-27T16:04:36.3286232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjw3tcpz/_remote_module_non_scriptable.py 2022-09-27T16:04:36.5846114Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:36.5856682Z 2022-09-27T16:04:36.5857097Z Running tests... 2022-09-27T16:04:36.5857538Z ---------------------------------------------------------------------- 2022-09-27T16:04:36.8238754Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1922 2022-09-27T16:04:36.8264761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1923 2022-09-27T16:04:36.8292065Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1924 2022-09-27T16:04:36.8321964Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1925 2022-09-27T16:04:37.9692956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:37.9693650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:37.9763274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:37.9763764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:37.9797930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:37.9798504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:38.0037118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:38.0037780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:38.0069875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:38.0070504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:38.0146649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:38.0147374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:38.0179389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:38.0180089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:38.0395188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:38.0395812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:38.1322219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_wfi06wj 2022-09-27T16:04:38.1323137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_wfi06wj/_remote_module_non_scriptable.py 2022-09-27T16:04:38.1397658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyy87qw44 2022-09-27T16:04:38.1398835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyy87qw44/_remote_module_non_scriptable.py 2022-09-27T16:04:38.1431642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3f1vpb5z 2022-09-27T16:04:38.1433387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3f1vpb5z/_remote_module_non_scriptable.py 2022-09-27T16:04:38.1650210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5ziz3sp 2022-09-27T16:04:38.1650965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5ziz3sp/_remote_module_non_scriptable.py 2022-09-27T16:04:38.3876279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:38.3961814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:38.4183930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:38.4269006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:38.7375301Z skip: Need at least 1 CUDA device (2.152s) 2022-09-27T16:04:38.7375564Z 2022-09-27T16:04:38.7375883Z ---------------------------------------------------------------------- 2022-09-27T16:04:38.7376151Z Ran 1 test in 2.152s 2022-09-27T16:04:38.7376333Z 2022-09-27T16:04:38.7376426Z OK (skipped=1) 2022-09-27T16:04:38.7376552Z 2022-09-27T16:04:38.7376638Z Generating XML reports... 2022-09-27T16:04:38.7420866Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160436.xml 2022-09-27T16:04:40.1102132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:40.1102570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:40.1110558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:40.1111030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:40.2706384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqw53p06s 2022-09-27T16:04:40.2706898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqw53p06s/_remote_module_non_scriptable.py 2022-09-27T16:04:40.5265336Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:40.5275699Z 2022-09-27T16:04:40.5275892Z Running tests... 2022-09-27T16:04:40.5276254Z ---------------------------------------------------------------------- 2022-09-27T16:04:40.7730307Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1977 2022-09-27T16:04:40.7756713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1978 2022-09-27T16:04:40.7783816Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1979 2022-09-27T16:04:40.7811716Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1980 2022-09-27T16:04:41.8801757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:41.8802525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:41.8812628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:41.8813217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:41.9047006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:41.9047569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:41.9092074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:41.9092740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:41.9190458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:41.9191276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:41.9202458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:41.9203039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:41.9426627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:41.9427278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:41.9465821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:41.9466491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:42.0468773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12xbmpft 2022-09-27T16:04:42.0469542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1l5f4au 2022-09-27T16:04:42.0470278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12xbmpft/_remote_module_non_scriptable.py 2022-09-27T16:04:42.0471088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1l5f4au/_remote_module_non_scriptable.py 2022-09-27T16:04:42.0679481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdj07sew 2022-09-27T16:04:42.0680793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdj07sew/_remote_module_non_scriptable.py 2022-09-27T16:04:42.0715471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50kyo5v3 2022-09-27T16:04:42.0716590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50kyo5v3/_remote_module_non_scriptable.py 2022-09-27T16:04:42.3007924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:42.3026930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:42.3224989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:42.3255595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:42.6866517Z skip: Need at least 1 CUDA device (2.159s) 2022-09-27T16:04:42.6866847Z 2022-09-27T16:04:42.6867275Z ---------------------------------------------------------------------- 2022-09-27T16:04:42.6867529Z Ran 1 test in 2.159s 2022-09-27T16:04:42.6867644Z 2022-09-27T16:04:42.6867717Z OK (skipped=1) 2022-09-27T16:04:42.6867832Z 2022-09-27T16:04:42.6867918Z Generating XML reports... 2022-09-27T16:04:42.6912055Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160440.xml 2022-09-27T16:04:44.0407363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:44.0407760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:44.0415115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:44.0415476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:44.1974597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4dmzziv2 2022-09-27T16:04:44.1975322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4dmzziv2/_remote_module_non_scriptable.py 2022-09-27T16:04:44.4549270Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:44.4560147Z 2022-09-27T16:04:44.4560224Z Running tests... 2022-09-27T16:04:44.4561192Z ---------------------------------------------------------------------- 2022-09-27T16:04:44.7021515Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2032 2022-09-27T16:04:44.7048643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2033 2022-09-27T16:04:44.7076875Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2034 2022-09-27T16:04:44.7113579Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2035 2022-09-27T16:04:45.8593880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:45.8594772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:45.8595575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:45.8596205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:45.8979754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:45.8980348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:45.8981086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:45.8981654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:45.9307914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:45.9308533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:45.9433707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:45.9434325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:45.9683844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:45.9684514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:45.9808916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:45.9809546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:46.0262118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tvd9516 2022-09-27T16:04:46.0262768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9tav63q 2022-09-27T16:04:46.0263410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tvd9516/_remote_module_non_scriptable.py 2022-09-27T16:04:46.0264088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9tav63q/_remote_module_non_scriptable.py 2022-09-27T16:04:46.0952236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplldzf5o5 2022-09-27T16:04:46.0953213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplldzf5o5/_remote_module_non_scriptable.py 2022-09-27T16:04:46.1071313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzwb7ct55 2022-09-27T16:04:46.1072457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzwb7ct55/_remote_module_non_scriptable.py 2022-09-27T16:04:46.2810445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:46.2813879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:46.3525969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:46.3647655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:46.7168726Z skip: Need at least 2 CUDA devices (2.261s) 2022-09-27T16:04:46.7175605Z 2022-09-27T16:04:46.7176521Z ---------------------------------------------------------------------- 2022-09-27T16:04:46.7176956Z Ran 1 test in 2.261s 2022-09-27T16:04:46.7177135Z 2022-09-27T16:04:46.7177262Z OK (skipped=1) 2022-09-27T16:04:46.7177448Z 2022-09-27T16:04:46.7177588Z Generating XML reports... 2022-09-27T16:04:46.7217246Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160444.xml 2022-09-27T16:04:48.0930518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:48.0931173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:48.0939880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:48.0940500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:48.2507743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnuzmiu36 2022-09-27T16:04:48.2508623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnuzmiu36/_remote_module_non_scriptable.py 2022-09-27T16:04:48.5106023Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:48.5116778Z 2022-09-27T16:04:48.5116915Z Running tests... 2022-09-27T16:04:48.5117471Z ---------------------------------------------------------------------- 2022-09-27T16:04:48.7533764Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79750 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.241s) 2022-09-27T16:04:48.7534388Z 2022-09-27T16:04:48.7534597Z ---------------------------------------------------------------------- 2022-09-27T16:04:48.7534851Z Ran 1 test in 0.241s 2022-09-27T16:04:48.7534980Z 2022-09-27T16:04:48.7535051Z OK (skipped=1) 2022-09-27T16:04:48.7535160Z 2022-09-27T16:04:48.7535242Z Generating XML reports... 2022-09-27T16:04:48.7564491Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160448.xml 2022-09-27T16:04:50.0549102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:50.0549484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:50.0557753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:50.0558125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:50.2137961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp185b2bch 2022-09-27T16:04:50.2138842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp185b2bch/_remote_module_non_scriptable.py 2022-09-27T16:04:50.4722410Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:50.4733334Z 2022-09-27T16:04:50.4733644Z Running tests... 2022-09-27T16:04:50.4734302Z ---------------------------------------------------------------------- 2022-09-27T16:04:50.7117706Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2097 2022-09-27T16:04:50.7144029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2098 2022-09-27T16:04:50.7171357Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2099 2022-09-27T16:04:50.7199915Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2100 2022-09-27T16:04:51.8584338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:51.8584976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:51.8664157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:51.8664719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:51.8916783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:51.8917281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:51.8953574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:51.8954237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:51.8984638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:51.8985330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:51.9041042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:51.9041569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:51.9314993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:51.9315571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:51.9378128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:51.9378750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:52.0228743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpdxtm_v 2022-09-27T16:04:52.0229574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpdxtm_v/_remote_module_non_scriptable.py 2022-09-27T16:04:52.0304968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhu9b80j 2022-09-27T16:04:52.0306161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhu9b80j/_remote_module_non_scriptable.py 2022-09-27T16:04:52.0578738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdlsctpgg 2022-09-27T16:04:52.0579553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdlsctpgg/_remote_module_non_scriptable.py 2022-09-27T16:04:52.0635493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfz31_1ay 2022-09-27T16:04:52.0636395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfz31_1ay/_remote_module_non_scriptable.py 2022-09-27T16:04:52.2806335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:52.2858626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:52.3130484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:52.3207022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:52.6253795Z skip: Need at least 2 CUDA devices (2.152s) 2022-09-27T16:04:52.6254058Z 2022-09-27T16:04:52.6254512Z ---------------------------------------------------------------------- 2022-09-27T16:04:52.6254899Z Ran 1 test in 2.152s 2022-09-27T16:04:52.6255073Z 2022-09-27T16:04:52.6255185Z OK (skipped=1) 2022-09-27T16:04:52.6255351Z 2022-09-27T16:04:52.6255481Z Generating XML reports... 2022-09-27T16:04:52.6299980Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160450.xml 2022-09-27T16:04:53.9897200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:53.9897669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:53.9907048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:53.9907505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:54.1513535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_n_jfnfd 2022-09-27T16:04:54.1514736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_n_jfnfd/_remote_module_non_scriptable.py 2022-09-27T16:04:54.4068855Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:54.4080144Z 2022-09-27T16:04:54.4080465Z Running tests... 2022-09-27T16:04:54.4081114Z ---------------------------------------------------------------------- 2022-09-27T16:04:54.6474960Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2152 2022-09-27T16:04:54.6501399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2153 2022-09-27T16:04:54.6528549Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2154 2022-09-27T16:04:54.6558334Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2155 2022-09-27T16:04:55.8225345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:55.8225994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:55.8567178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:55.8567814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:55.8672768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:55.8673461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:55.8784323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:55.8784838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:55.8790437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:55.8791169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:55.9052792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:55.9053452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:55.9166674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:55.9167308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:55.9173936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:55.9174677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:56.0028004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl12x916a 2022-09-27T16:04:56.0028798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl12x916a/_remote_module_non_scriptable.py 2022-09-27T16:04:56.0338402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqv3a73vt 2022-09-27T16:04:56.0339173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqv3a73vt/_remote_module_non_scriptable.py 2022-09-27T16:04:56.0442136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp155n_4s_ 2022-09-27T16:04:56.0443268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp155n_4s_/_remote_module_non_scriptable.py 2022-09-27T16:04:56.0443968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ocpyr9y 2022-09-27T16:04:56.0447340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ocpyr9y/_remote_module_non_scriptable.py 2022-09-27T16:04:56.2577308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:04:56.2870537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:04:56.2994928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:04:56.3065648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:04:56.6612794Z skip: Need at least 2 CUDA devices (2.253s) 2022-09-27T16:04:56.6613053Z 2022-09-27T16:04:56.6613477Z ---------------------------------------------------------------------- 2022-09-27T16:04:56.6613920Z Ran 1 test in 2.253s 2022-09-27T16:04:56.6614132Z 2022-09-27T16:04:56.6614264Z OK (skipped=1) 2022-09-27T16:04:56.6614440Z 2022-09-27T16:04:56.6614528Z Generating XML reports... 2022-09-27T16:04:56.6660426Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160454.xml 2022-09-27T16:04:58.0360221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:58.0360592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:58.0369216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:58.0369577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:58.1958330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5uh_kbm 2022-09-27T16:04:58.1959182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5uh_kbm/_remote_module_non_scriptable.py 2022-09-27T16:04:58.4540582Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:04:58.4551854Z 2022-09-27T16:04:58.4552281Z Running tests... 2022-09-27T16:04:58.4552918Z ---------------------------------------------------------------------- 2022-09-27T16:04:58.6959334Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2207 2022-09-27T16:04:58.6987051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2208 2022-09-27T16:04:58.7014347Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2209 2022-09-27T16:04:58.7044214Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2210 2022-09-27T16:04:59.8026084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:59.8027040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:59.8381253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:59.8382274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:59.8555987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:59.8556701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:59.8698172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:59.8698649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:59.8753168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:04:59.8753856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:04:59.8952304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:59.8952965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:59.9094907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:59.9095620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:59.9146364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:04:59.9147027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:04:59.9666039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv98k0iub 2022-09-27T16:04:59.9666746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv98k0iub/_remote_module_non_scriptable.py 2022-09-27T16:05:00.0218814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp91lrrea7 2022-09-27T16:05:00.0219612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp91lrrea7/_remote_module_non_scriptable.py 2022-09-27T16:05:00.0355314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vhayaur 2022-09-27T16:05:00.0356417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vhayaur/_remote_module_non_scriptable.py 2022-09-27T16:05:00.0412410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqef9xk1y 2022-09-27T16:05:00.0413352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqef9xk1y/_remote_module_non_scriptable.py 2022-09-27T16:05:00.2243046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:00.2783333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:00.3011558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:00.3130670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:00.6097859Z skip: Need at least 2 CUDA devices (2.154s) 2022-09-27T16:05:00.6098086Z 2022-09-27T16:05:00.6098402Z ---------------------------------------------------------------------- 2022-09-27T16:05:00.6098710Z Ran 1 test in 2.155s 2022-09-27T16:05:00.6098827Z 2022-09-27T16:05:00.6098899Z OK (skipped=1) 2022-09-27T16:05:00.6099010Z 2022-09-27T16:05:00.6099096Z Generating XML reports... 2022-09-27T16:05:00.6142345Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160458.xml 2022-09-27T16:05:02.0178544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:02.0179186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:02.0188731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:02.0189379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:02.1898698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf78b8pbo 2022-09-27T16:05:02.1899431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf78b8pbo/_remote_module_non_scriptable.py 2022-09-27T16:05:02.4496948Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:02.4507290Z 2022-09-27T16:05:02.4507640Z Running tests... 2022-09-27T16:05:02.4508264Z ---------------------------------------------------------------------- 2022-09-27T16:05:02.6932661Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2262 2022-09-27T16:05:02.6960843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2263 2022-09-27T16:05:02.6988877Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2264 2022-09-27T16:05:02.7018797Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2265 2022-09-27T16:05:03.8196251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:03.8196893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:03.8533703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:03.8534421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:03.8563757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:03.8564403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:03.8574783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:03.8575393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:03.8775076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:03.8775709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:03.8922425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:03.8923030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:03.8963334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:03.8963975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:03.9153815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:03.9154458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:03.9832600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyo5x5wyj 2022-09-27T16:05:03.9833405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyo5x5wyj/_remote_module_non_scriptable.py 2022-09-27T16:05:04.0205448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6797aj7 2022-09-27T16:05:04.0206732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6797aj7/_remote_module_non_scriptable.py 2022-09-27T16:05:04.0222886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbol4wwqi 2022-09-27T16:05:04.0224843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbol4wwqi/_remote_module_non_scriptable.py 2022-09-27T16:05:04.0419143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12bbyike 2022-09-27T16:05:04.0420191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12bbyike/_remote_module_non_scriptable.py 2022-09-27T16:05:04.2412619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:04.2795572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:04.2976111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:04.2983313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:05.0080038Z ok (2.557s) 2022-09-27T16:05:05.0080293Z 2022-09-27T16:05:05.0080799Z ---------------------------------------------------------------------- 2022-09-27T16:05:05.0081218Z Ran 1 test in 2.557s 2022-09-27T16:05:05.0081423Z 2022-09-27T16:05:05.0081533Z OK 2022-09-27T16:05:05.0081704Z 2022-09-27T16:05:05.0081876Z Generating XML reports... 2022-09-27T16:05:05.0127022Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160502.xml 2022-09-27T16:05:06.4148190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:06.4148556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:06.4157264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:06.4157630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:06.5744635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_bvmb9s 2022-09-27T16:05:06.5746197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_bvmb9s/_remote_module_non_scriptable.py 2022-09-27T16:05:06.8341786Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:06.8352170Z 2022-09-27T16:05:06.8352299Z Running tests... 2022-09-27T16:05:06.8353175Z ---------------------------------------------------------------------- 2022-09-27T16:05:07.0813229Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2485 2022-09-27T16:05:07.0839474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2486 2022-09-27T16:05:07.0866822Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2487 2022-09-27T16:05:07.0895812Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2488 2022-09-27T16:05:08.2622445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:08.2623101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:08.2977134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:08.2977808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:08.3022563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:08.3023158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:08.3364845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:08.3365501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:08.3440343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:08.3440981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:08.3801312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:08.3802269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:08.3911668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:08.3912627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:08.4278812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:08.4279502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:08.4436912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp22r4vhp 2022-09-27T16:05:08.4437733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp22r4vhp/_remote_module_non_scriptable.py 2022-09-27T16:05:08.4653534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjlp_jwq 2022-09-27T16:05:08.4654246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjlp_jwq/_remote_module_non_scriptable.py 2022-09-27T16:05:08.5071445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppb2eexca 2022-09-27T16:05:08.5072298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppb2eexca/_remote_module_non_scriptable.py 2022-09-27T16:05:08.5548145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2krzhb5v 2022-09-27T16:05:08.5548956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2krzhb5v/_remote_module_non_scriptable.py 2022-09-27T16:05:08.7031779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:08.7263328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:08.7660228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:08.8124680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:09.0951410Z skip: Need at least 1 CUDA device (2.260s) 2022-09-27T16:05:09.0951755Z 2022-09-27T16:05:09.0952265Z ---------------------------------------------------------------------- 2022-09-27T16:05:09.0952585Z Ran 1 test in 2.260s 2022-09-27T16:05:09.0952701Z 2022-09-27T16:05:09.0952773Z OK (skipped=1) 2022-09-27T16:05:09.0952881Z 2022-09-27T16:05:09.0952952Z Generating XML reports... 2022-09-27T16:05:09.0998509Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160506.xml 2022-09-27T16:05:10.4888051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:10.4888435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:10.4896768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:10.4897253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:10.6478825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5w4pwn6w 2022-09-27T16:05:10.6479788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5w4pwn6w/_remote_module_non_scriptable.py 2022-09-27T16:05:10.9055805Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:10.9066753Z 2022-09-27T16:05:10.9067076Z Running tests... 2022-09-27T16:05:10.9067490Z ---------------------------------------------------------------------- 2022-09-27T16:05:11.1460895Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2540 2022-09-27T16:05:11.1487970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2541 2022-09-27T16:05:11.1515270Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2542 2022-09-27T16:05:11.1543499Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2543 2022-09-27T16:05:12.3167160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:12.3167832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:12.3443892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:12.3444495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:12.3555454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:12.3556108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:12.3639657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:12.3640290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:12.3751176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:12.3751768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:12.3835030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:12.3835589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:12.4026478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:12.4027124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:12.4125297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:12.4125977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:12.4843060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwl0gat8q 2022-09-27T16:05:12.4843855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwl0gat8q/_remote_module_non_scriptable.py 2022-09-27T16:05:12.5116623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8r6l0dox 2022-09-27T16:05:12.5117896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8r6l0dox/_remote_module_non_scriptable.py 2022-09-27T16:05:12.5302996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3jawb34 2022-09-27T16:05:12.5304047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3jawb34/_remote_module_non_scriptable.py 2022-09-27T16:05:12.5585359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5z_n2s5 2022-09-27T16:05:12.5586775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5z_n2s5/_remote_module_non_scriptable.py 2022-09-27T16:05:12.7479985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:12.7714912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:12.7922542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:12.8195457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:13.1599761Z skip: Need at least 2 CUDA devices (2.253s) 2022-09-27T16:05:13.1600045Z 2022-09-27T16:05:13.1600556Z ---------------------------------------------------------------------- 2022-09-27T16:05:13.1601003Z Ran 1 test in 2.253s 2022-09-27T16:05:13.1601204Z 2022-09-27T16:05:13.1601336Z OK (skipped=1) 2022-09-27T16:05:13.1601521Z 2022-09-27T16:05:13.1601658Z Generating XML reports... 2022-09-27T16:05:13.1645282Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160510.xml 2022-09-27T16:05:14.5228128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:14.5228751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:14.5236123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:14.5236495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:14.6807946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6d92jx49 2022-09-27T16:05:14.6809197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6d92jx49/_remote_module_non_scriptable.py 2022-09-27T16:05:14.9376693Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:14.9387312Z 2022-09-27T16:05:14.9387433Z Running tests... 2022-09-27T16:05:14.9387963Z ---------------------------------------------------------------------- 2022-09-27T16:05:15.1800169Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2595 2022-09-27T16:05:15.1826917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2596 2022-09-27T16:05:15.1854491Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2597 2022-09-27T16:05:15.1883653Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2598 2022-09-27T16:05:16.2809357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:16.2810073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:16.2870370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:16.2870977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:16.2892896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:16.2893599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:16.2894387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:16.2895017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:16.3182120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:16.3182781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:16.3251115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:16.3251733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:16.3270508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:16.3271375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:16.3272321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:16.3273016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:16.4461400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvwzdui8 2022-09-27T16:05:16.4462197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvwzdui8/_remote_module_non_scriptable.py 2022-09-27T16:05:16.4516945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpouolh2vr 2022-09-27T16:05:16.4518546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpouolh2vr/_remote_module_non_scriptable.py 2022-09-27T16:05:16.4538579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9b9v2w4w 2022-09-27T16:05:16.4540033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9b9v2w4w/_remote_module_non_scriptable.py 2022-09-27T16:05:16.4546247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3mknprz 2022-09-27T16:05:16.4548512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3mknprz/_remote_module_non_scriptable.py 2022-09-27T16:05:16.7054312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:16.7079815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:16.7120188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:16.7234399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:16.9935531Z skip: Need at least 2 CUDA devices (2.054s) 2022-09-27T16:05:16.9935827Z 2022-09-27T16:05:16.9936271Z ---------------------------------------------------------------------- 2022-09-27T16:05:16.9936515Z Ran 1 test in 2.055s 2022-09-27T16:05:16.9936629Z 2022-09-27T16:05:16.9936702Z OK (skipped=1) 2022-09-27T16:05:16.9936807Z 2022-09-27T16:05:16.9936892Z Generating XML reports... 2022-09-27T16:05:16.9979705Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160514.xml 2022-09-27T16:05:18.3637472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:18.3637853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:18.3646038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:18.3646401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:18.5204418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqskb64zs 2022-09-27T16:05:18.5205402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqskb64zs/_remote_module_non_scriptable.py 2022-09-27T16:05:18.7789082Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:18.7799798Z 2022-09-27T16:05:18.7799889Z Running tests... 2022-09-27T16:05:18.7800607Z ---------------------------------------------------------------------- 2022-09-27T16:05:19.0113023Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/80008 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.231s) 2022-09-27T16:05:19.0113610Z 2022-09-27T16:05:19.0113835Z ---------------------------------------------------------------------- 2022-09-27T16:05:19.0114078Z Ran 1 test in 0.231s 2022-09-27T16:05:19.0114200Z 2022-09-27T16:05:19.0114259Z OK (skipped=1) 2022-09-27T16:05:19.0114369Z 2022-09-27T16:05:19.0114453Z Generating XML reports... 2022-09-27T16:05:19.0142680Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160518.xml 2022-09-27T16:05:20.3483430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:20.3483860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:20.3491477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:20.3491832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:20.5051594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3d_fnur0 2022-09-27T16:05:20.5052055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3d_fnur0/_remote_module_non_scriptable.py 2022-09-27T16:05:20.7604866Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:20.7614952Z 2022-09-27T16:05:20.7615095Z Running tests... 2022-09-27T16:05:20.7615545Z ---------------------------------------------------------------------- 2022-09-27T16:05:21.0052328Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2660 2022-09-27T16:05:21.0079684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2661 2022-09-27T16:05:21.0106935Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2662 2022-09-27T16:05:21.0136354Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2663 2022-09-27T16:05:22.1605385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:22.1605988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:22.1814718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:22.1815425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:22.1968282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:22.1968799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:22.1999229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:22.1999740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:22.2185979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:22.2186601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:22.2216410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:22.2217006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:22.2368282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:22.2368796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:22.2579662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:22.2580353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:22.3249988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkx5hycyn 2022-09-27T16:05:22.3251027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkx5hycyn/_remote_module_non_scriptable.py 2022-09-27T16:05:22.3455024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9yzz70hy 2022-09-27T16:05:22.3455845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9yzz70hy/_remote_module_non_scriptable.py 2022-09-27T16:05:22.3641010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_pxfqs5 2022-09-27T16:05:22.3642234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_pxfqs5/_remote_module_non_scriptable.py 2022-09-27T16:05:22.3842625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwjqxg8ik 2022-09-27T16:05:22.3843549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwjqxg8ik/_remote_module_non_scriptable.py 2022-09-27T16:05:22.5834852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:22.6016068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:22.6242495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:22.6627229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:23.0192431Z skip: Need at least 2 CUDA devices (2.257s) 2022-09-27T16:05:23.0192758Z 2022-09-27T16:05:23.0193282Z ---------------------------------------------------------------------- 2022-09-27T16:05:23.0193615Z Ran 1 test in 2.258s 2022-09-27T16:05:23.0193737Z 2022-09-27T16:05:23.0193811Z OK (skipped=1) 2022-09-27T16:05:23.0193908Z 2022-09-27T16:05:23.0193997Z Generating XML reports... 2022-09-27T16:05:23.0238888Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160520.xml 2022-09-27T16:05:24.4059168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:24.4059545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:24.4067135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:24.4067500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:24.5631997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmscsjsa 2022-09-27T16:05:24.5632631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmscsjsa/_remote_module_non_scriptable.py 2022-09-27T16:05:24.8225734Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:24.8236229Z 2022-09-27T16:05:24.8236342Z Running tests... 2022-09-27T16:05:24.8236793Z ---------------------------------------------------------------------- 2022-09-27T16:05:25.0649319Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2715 2022-09-27T16:05:25.0676445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2716 2022-09-27T16:05:25.0703042Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2717 2022-09-27T16:05:25.0731775Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2718 2022-09-27T16:05:26.2265878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:26.2266531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:26.2267272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:26.2267823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:26.2581855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:26.2582517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:26.2651899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:26.2652559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:26.2653353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:26.2653952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:26.2944701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:26.2945386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:26.2957967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:26.2958638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:26.3320445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:26.3321113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:26.3925199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkiu_ygt4 2022-09-27T16:05:26.3926158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb3ja_6qu 2022-09-27T16:05:26.3926839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkiu_ygt4/_remote_module_non_scriptable.py 2022-09-27T16:05:26.3927512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb3ja_6qu/_remote_module_non_scriptable.py 2022-09-27T16:05:26.4199992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5qjx3sfg 2022-09-27T16:05:26.4201194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5qjx3sfg/_remote_module_non_scriptable.py 2022-09-27T16:05:26.4587781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjof9i4r 2022-09-27T16:05:26.4588605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjof9i4r/_remote_module_non_scriptable.py 2022-09-27T16:05:26.6484701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:26.6494546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:26.6755076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:26.7179750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:26.9841427Z skip: Need at least 2 CUDA devices (2.160s) 2022-09-27T16:05:26.9841714Z 2022-09-27T16:05:26.9842199Z ---------------------------------------------------------------------- 2022-09-27T16:05:26.9842469Z Ran 1 test in 2.160s 2022-09-27T16:05:26.9842583Z 2022-09-27T16:05:26.9842656Z OK (skipped=1) 2022-09-27T16:05:26.9842771Z 2022-09-27T16:05:26.9842843Z Generating XML reports... 2022-09-27T16:05:26.9885992Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160524.xml 2022-09-27T16:05:28.3494831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:28.3495214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:28.3503632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:28.3503987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:28.5117135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59af4spk 2022-09-27T16:05:28.5118120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59af4spk/_remote_module_non_scriptable.py 2022-09-27T16:05:28.7718340Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:28.7729495Z 2022-09-27T16:05:28.7729761Z Running tests... 2022-09-27T16:05:28.7730167Z ---------------------------------------------------------------------- 2022-09-27T16:05:29.0154221Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2770 2022-09-27T16:05:29.0180053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2771 2022-09-27T16:05:29.0207142Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2772 2022-09-27T16:05:29.0235735Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2773 2022-09-27T16:05:30.0893861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:30.0894302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:30.1022621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:30.1023288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:30.1271322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:30.1271961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:30.1279468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:30.1280092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:30.1391008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:30.1391561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:30.1497537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:30.1498174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:30.1653915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:30.1654558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:30.1863547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:30.1864233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:30.2547692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3ojw3um 2022-09-27T16:05:30.2548503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3ojw3um/_remote_module_non_scriptable.py 2022-09-27T16:05:30.2649874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyut3ujl1 2022-09-27T16:05:30.2650947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyut3ujl1/_remote_module_non_scriptable.py 2022-09-27T16:05:30.2912702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuhmn9ut7 2022-09-27T16:05:30.2914112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuhmn9ut7/_remote_module_non_scriptable.py 2022-09-27T16:05:30.3121413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmm0plagu 2022-09-27T16:05:30.3122484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmm0plagu/_remote_module_non_scriptable.py 2022-09-27T16:05:30.5111247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:30.5210736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:30.5561284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:30.5702686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:30.9289840Z skip: Need at least 2 CUDA devices (2.156s) 2022-09-27T16:05:30.9290160Z 2022-09-27T16:05:30.9290646Z ---------------------------------------------------------------------- 2022-09-27T16:05:30.9290899Z Ran 1 test in 2.156s 2022-09-27T16:05:30.9291037Z 2022-09-27T16:05:30.9291110Z OK (skipped=1) 2022-09-27T16:05:30.9291217Z 2022-09-27T16:05:30.9291301Z Generating XML reports... 2022-09-27T16:05:30.9337525Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160528.xml 2022-09-27T16:05:32.3050837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:32.3051263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:32.3059303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:32.3059768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:32.4627608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp470tk5l7 2022-09-27T16:05:32.4628327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp470tk5l7/_remote_module_non_scriptable.py 2022-09-27T16:05:32.7187238Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:32.7198245Z 2022-09-27T16:05:32.7198636Z Running tests... 2022-09-27T16:05:32.7199022Z ---------------------------------------------------------------------- 2022-09-27T16:05:32.9590834Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2825 2022-09-27T16:05:32.9617968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2826 2022-09-27T16:05:32.9645472Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2827 2022-09-27T16:05:32.9674758Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2828 2022-09-27T16:05:34.0873230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:34.0873893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:34.1206282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:34.1206991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:34.1237792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:34.1238372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:34.1411995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:34.1412611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:34.1581148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:34.1581808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:34.1668429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:34.1669045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:34.1788142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:34.1788814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:34.2037067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:34.2037710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:34.2528881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17_3n64z 2022-09-27T16:05:34.2529768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17_3n64z/_remote_module_non_scriptable.py 2022-09-27T16:05:34.3040356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfs9er_y3 2022-09-27T16:05:34.3041274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfs9er_y3/_remote_module_non_scriptable.py 2022-09-27T16:05:34.3089731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_90ne22 2022-09-27T16:05:34.3091271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_90ne22/_remote_module_non_scriptable.py 2022-09-27T16:05:34.3309243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_xai1em 2022-09-27T16:05:34.3310206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_xai1em/_remote_module_non_scriptable.py 2022-09-27T16:05:34.5153865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:34.5651557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:34.5687224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:34.6040832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:34.9728558Z skip: Need at least 2 CUDA devices (2.253s) 2022-09-27T16:05:34.9728825Z 2022-09-27T16:05:34.9729320Z ---------------------------------------------------------------------- 2022-09-27T16:05:34.9729606Z Ran 1 test in 2.253s 2022-09-27T16:05:34.9729729Z 2022-09-27T16:05:34.9729802Z OK (skipped=1) 2022-09-27T16:05:34.9729910Z 2022-09-27T16:05:34.9729995Z Generating XML reports... 2022-09-27T16:05:34.9773242Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160532.xml 2022-09-27T16:05:36.3219897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:36.3220271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:36.3227954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:36.3228362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:36.4773948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6khkfr_6 2022-09-27T16:05:36.4774892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6khkfr_6/_remote_module_non_scriptable.py 2022-09-27T16:05:36.7351941Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:36.7361890Z 2022-09-27T16:05:36.7362021Z Running tests... 2022-09-27T16:05:36.7362584Z ---------------------------------------------------------------------- 2022-09-27T16:05:36.9853598Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2880 2022-09-27T16:05:36.9881469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2881 2022-09-27T16:05:36.9909227Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2882 2022-09-27T16:05:36.9938444Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2883 2022-09-27T16:05:38.0764641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:38.0765282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:38.0916307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:38.0917002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:38.0993762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:38.0994276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:38.1142452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:38.1142995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:38.1279482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:38.1280495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:38.1302340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:38.1303170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:38.1371185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:38.1371849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:38.1638756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:38.1639391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:38.2412737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkunbot4 2022-09-27T16:05:38.2413747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkunbot4/_remote_module_non_scriptable.py 2022-09-27T16:05:38.2584394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_4_3gsi 2022-09-27T16:05:38.2586065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_4_3gsi/_remote_module_non_scriptable.py 2022-09-27T16:05:38.2644595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprulqjd9j 2022-09-27T16:05:38.2646033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprulqjd9j/_remote_module_non_scriptable.py 2022-09-27T16:05:38.2908520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjmpv6z8 2022-09-27T16:05:38.2909482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjmpv6z8/_remote_module_non_scriptable.py 2022-09-27T16:05:38.5018466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:38.5331825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:38.5387076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:38.5529747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:38.8993289Z skip: Need at least 2 CUDA devices (2.163s) 2022-09-27T16:05:38.8993556Z 2022-09-27T16:05:38.8993964Z ---------------------------------------------------------------------- 2022-09-27T16:05:38.8994218Z Ran 1 test in 2.163s 2022-09-27T16:05:38.8994335Z 2022-09-27T16:05:38.8994410Z OK (skipped=1) 2022-09-27T16:05:38.8994625Z 2022-09-27T16:05:38.8994753Z Generating XML reports... 2022-09-27T16:05:38.9040749Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160536.xml 2022-09-27T16:05:40.2690690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:40.2691067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:40.2700516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:40.2700975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:40.4339750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnc6abwow 2022-09-27T16:05:40.4340650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnc6abwow/_remote_module_non_scriptable.py 2022-09-27T16:05:40.6987236Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:40.6997089Z 2022-09-27T16:05:40.6997247Z Running tests... 2022-09-27T16:05:40.6997692Z ---------------------------------------------------------------------- 2022-09-27T16:05:40.9408401Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2935 2022-09-27T16:05:40.9435970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2936 2022-09-27T16:05:40.9462855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2937 2022-09-27T16:05:40.9491671Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2938 2022-09-27T16:05:42.0688026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:42.0688715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:42.0704568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:42.0705227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:42.0772687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:42.0773211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:42.0824100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:42.0824614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:42.1070528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:42.1071168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:42.1088650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:42.1089265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:42.1151336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:42.1151931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:42.1197412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:42.1198049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:42.2337414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9zomkjm 2022-09-27T16:05:42.2338213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9zomkjm/_remote_module_non_scriptable.py 2022-09-27T16:05:42.2347803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmocan4w9 2022-09-27T16:05:42.2349948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmocan4w9/_remote_module_non_scriptable.py 2022-09-27T16:05:42.2413875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9o1xu6u2 2022-09-27T16:05:42.2415779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9o1xu6u2/_remote_module_non_scriptable.py 2022-09-27T16:05:42.2450270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0uhziztu 2022-09-27T16:05:42.2451560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0uhziztu/_remote_module_non_scriptable.py 2022-09-27T16:05:42.4901280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:42.4902555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:42.5030025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:42.5049774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:42.8546151Z skip: Need at least 2 CUDA devices (2.155s) 2022-09-27T16:05:42.8546474Z 2022-09-27T16:05:42.8546981Z ---------------------------------------------------------------------- 2022-09-27T16:05:42.8547261Z Ran 1 test in 2.155s 2022-09-27T16:05:42.8547376Z 2022-09-27T16:05:42.8547447Z OK (skipped=1) 2022-09-27T16:05:42.8547798Z 2022-09-27T16:05:42.8547886Z Generating XML reports... 2022-09-27T16:05:42.8591983Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160540.xml 2022-09-27T16:05:44.2400924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:44.2401316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:44.2408401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:44.2408763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:44.3972063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqzwurw8 2022-09-27T16:05:44.3973516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqzwurw8/_remote_module_non_scriptable.py 2022-09-27T16:05:44.6544783Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:44.6555457Z 2022-09-27T16:05:44.6555655Z Running tests... 2022-09-27T16:05:44.6556060Z ---------------------------------------------------------------------- 2022-09-27T16:05:44.8974144Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2990 2022-09-27T16:05:44.9000272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2991 2022-09-27T16:05:44.9027618Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2992 2022-09-27T16:05:44.9057214Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2993 2022-09-27T16:05:46.0459855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:46.0460552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:46.0759727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:46.0760390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:46.0854663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:46.0855261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:46.0918209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:46.0918790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:46.1010813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:46.1011435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:46.1158497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:46.1159160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:46.1320741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:46.1321425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:46.1404507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:46.1405165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:46.2146893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpisi_p93m 2022-09-27T16:05:46.2147713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpisi_p93m/_remote_module_non_scriptable.py 2022-09-27T16:05:46.2419938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6clwx8cl 2022-09-27T16:05:46.2421113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6clwx8cl/_remote_module_non_scriptable.py 2022-09-27T16:05:46.2588284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphzq4ndgj 2022-09-27T16:05:46.2589100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphzq4ndgj/_remote_module_non_scriptable.py 2022-09-27T16:05:46.2663279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyu56_bc 2022-09-27T16:05:46.2664113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyu56_bc/_remote_module_non_scriptable.py 2022-09-27T16:05:46.4782169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:46.5013357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:46.5252201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:46.5259612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:46.8110707Z skip: Need at least 2 CUDA devices (2.155s) 2022-09-27T16:05:46.8110998Z 2022-09-27T16:05:46.8111302Z ---------------------------------------------------------------------- 2022-09-27T16:05:46.8111608Z Ran 1 test in 2.155s 2022-09-27T16:05:46.8111725Z 2022-09-27T16:05:46.8111785Z OK (skipped=1) 2022-09-27T16:05:46.8111891Z 2022-09-27T16:05:46.8111975Z Generating XML reports... 2022-09-27T16:05:46.8157322Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160544.xml 2022-09-27T16:05:48.1974076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:48.1974460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:48.1982257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:48.1982811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:48.3567936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy049njv6 2022-09-27T16:05:48.3569301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy049njv6/_remote_module_non_scriptable.py 2022-09-27T16:05:48.6134715Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:48.6145537Z 2022-09-27T16:05:48.6145640Z Running tests... 2022-09-27T16:05:48.6146163Z ---------------------------------------------------------------------- 2022-09-27T16:05:48.8586426Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3045 2022-09-27T16:05:48.8614156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3046 2022-09-27T16:05:48.8641479Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3047 2022-09-27T16:05:48.8670639Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3048 2022-09-27T16:05:49.9928201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:49.9928836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:50.0229515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:50.0230119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:50.0294552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:50.0295137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:50.0333702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:50.0334222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:50.0604799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:50.0605422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:50.0694066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:50.0694738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:50.0705126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:50.0705712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:50.1062773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:50.1063369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:50.1558112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpalnxrmv9 2022-09-27T16:05:50.1559394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpalnxrmv9/_remote_module_non_scriptable.py 2022-09-27T16:05:50.1861713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjdsc1oad 2022-09-27T16:05:50.1862729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjdsc1oad/_remote_module_non_scriptable.py 2022-09-27T16:05:50.1949389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmple3fvnjw 2022-09-27T16:05:50.1950393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmple3fvnjw/_remote_module_non_scriptable.py 2022-09-27T16:05:50.2313143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbgp2qg2 2022-09-27T16:05:50.2313926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbgp2qg2/_remote_module_non_scriptable.py 2022-09-27T16:05:50.4130121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:50.4420218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:50.4522580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:50.4906011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:50.7724859Z skip: Need at least 2 CUDA devices (2.158s) 2022-09-27T16:05:50.7725168Z 2022-09-27T16:05:50.7725535Z ---------------------------------------------------------------------- 2022-09-27T16:05:50.7725804Z Ran 1 test in 2.158s 2022-09-27T16:05:50.7725917Z 2022-09-27T16:05:50.7725977Z OK (skipped=1) 2022-09-27T16:05:50.7726084Z 2022-09-27T16:05:50.7733264Z Generating XML reports... 2022-09-27T16:05:50.7771453Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160548.xml 2022-09-27T16:05:52.1705700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:52.1706068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:52.1714307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:52.1714865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:52.3268129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzlmqhcvb 2022-09-27T16:05:52.3269088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzlmqhcvb/_remote_module_non_scriptable.py 2022-09-27T16:05:52.5833143Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:52.5844052Z 2022-09-27T16:05:52.5844348Z Running tests... 2022-09-27T16:05:52.5844969Z ---------------------------------------------------------------------- 2022-09-27T16:05:52.8294939Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3100 2022-09-27T16:05:52.8335454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3101 2022-09-27T16:05:52.8368007Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3102 2022-09-27T16:05:52.8395193Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3103 2022-09-27T16:05:53.9439109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:53.9439776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:53.9546227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:53.9546763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:53.9827627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:53.9828200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:53.9849390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:53.9849997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:53.9941341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:53.9941999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:54.0196146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:54.0196829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:54.0256788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:54.0257333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:54.0595184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:54.0595811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:54.1114812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65dmvr_l 2022-09-27T16:05:54.1115643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65dmvr_l/_remote_module_non_scriptable.py 2022-09-27T16:05:54.1237117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0n73f7pm 2022-09-27T16:05:54.1238339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0n73f7pm/_remote_module_non_scriptable.py 2022-09-27T16:05:54.1524156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptooxc9vg 2022-09-27T16:05:54.1524966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptooxc9vg/_remote_module_non_scriptable.py 2022-09-27T16:05:54.1864863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2kauhdsq 2022-09-27T16:05:54.1865697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2kauhdsq/_remote_module_non_scriptable.py 2022-09-27T16:05:54.3700202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:54.3841754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:54.4113422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:54.4562243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:54.7449730Z skip: Need at least 2 CUDA devices (2.160s) 2022-09-27T16:05:54.7450290Z 2022-09-27T16:05:54.7450611Z ---------------------------------------------------------------------- 2022-09-27T16:05:54.7450981Z Ran 1 test in 2.160s 2022-09-27T16:05:54.7451130Z 2022-09-27T16:05:54.7451240Z OK (skipped=1) 2022-09-27T16:05:54.7451449Z 2022-09-27T16:05:54.7451609Z Generating XML reports... 2022-09-27T16:05:54.7493886Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160552.xml 2022-09-27T16:05:56.1119773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:56.1120459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:56.1128957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:56.1129577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:56.2693758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg6_aw2t9 2022-09-27T16:05:56.2694649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg6_aw2t9/_remote_module_non_scriptable.py 2022-09-27T16:05:56.5253140Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:05:56.5263974Z 2022-09-27T16:05:56.5264429Z Running tests... 2022-09-27T16:05:56.5264915Z ---------------------------------------------------------------------- 2022-09-27T16:05:56.7689454Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3155 2022-09-27T16:05:56.7716048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3156 2022-09-27T16:05:56.7743652Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3157 2022-09-27T16:05:56.7773089Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3158 2022-09-27T16:05:57.9397487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:57.9398154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:57.9530283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:57.9530790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:57.9531419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:57.9531873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:57.9686360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:05:57.9686982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:05:57.9779027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:57.9779587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:57.9913109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:57.9913660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:57.9921174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:57.9921765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:58.0060809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:05:58.0061731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:05:58.1179347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9li_fjjc 2022-09-27T16:05:58.1180323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9li_fjjc/_remote_module_non_scriptable.py 2022-09-27T16:05:58.1211560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp406_qli4 2022-09-27T16:05:58.1212455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp406_qli4/_remote_module_non_scriptable.py 2022-09-27T16:05:58.1219271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnx1zo1qy 2022-09-27T16:05:58.1221321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnx1zo1qy/_remote_module_non_scriptable.py 2022-09-27T16:05:58.1318567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4xzeyyx 2022-09-27T16:05:58.1320050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4xzeyyx/_remote_module_non_scriptable.py 2022-09-27T16:05:58.3730946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:05:58.3756765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:05:58.3810558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:05:58.3873860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:05:58.6826184Z skip: Need at least 2 CUDA devices (2.156s) 2022-09-27T16:05:58.6826491Z 2022-09-27T16:05:58.6826970Z ---------------------------------------------------------------------- 2022-09-27T16:05:58.6827272Z Ran 1 test in 2.156s 2022-09-27T16:05:58.6827385Z 2022-09-27T16:05:58.6827460Z OK (skipped=1) 2022-09-27T16:05:58.6827567Z 2022-09-27T16:05:58.6827652Z Generating XML reports... 2022-09-27T16:05:58.6871466Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160556.xml 2022-09-27T16:06:00.0685020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:00.0685405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:00.0694809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:00.0695172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:00.2293652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jv38cmj 2022-09-27T16:06:00.2294502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jv38cmj/_remote_module_non_scriptable.py 2022-09-27T16:06:00.4889554Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:00.4900430Z 2022-09-27T16:06:00.4900927Z Running tests... 2022-09-27T16:06:00.4901353Z ---------------------------------------------------------------------- 2022-09-27T16:06:00.7346129Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3210 2022-09-27T16:06:00.7373076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3211 2022-09-27T16:06:00.7400314Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3212 2022-09-27T16:06:00.7429253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3213 2022-09-27T16:06:01.8958171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:01.8958785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:01.9054205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:01.9054908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:01.9055707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:01.9056483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:01.9163228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:01.9163741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:01.9375981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:01.9376664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:01.9469506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:01.9470122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:01.9470932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:01.9471495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:01.9581972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:01.9582624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:02.0664552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbwvt9m5c 2022-09-27T16:06:02.0665727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbwvt9m5c/_remote_module_non_scriptable.py 2022-09-27T16:06:02.0743316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplxqnl4si 2022-09-27T16:06:02.0744245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplxqnl4si/_remote_module_non_scriptable.py 2022-09-27T16:06:02.0748467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30l4u1ag 2022-09-27T16:06:02.0750961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30l4u1ag/_remote_module_non_scriptable.py 2022-09-27T16:06:02.0844531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv4w4tusg 2022-09-27T16:06:02.0845732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv4w4tusg/_remote_module_non_scriptable.py 2022-09-27T16:06:02.3354721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:02.3366461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:02.3464012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:02.3542043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:02.6485134Z skip: Need at least 2 CUDA devices (2.158s) 2022-09-27T16:06:02.6485434Z 2022-09-27T16:06:02.6485945Z ---------------------------------------------------------------------- 2022-09-27T16:06:02.6486333Z Ran 1 test in 2.158s 2022-09-27T16:06:02.6486447Z 2022-09-27T16:06:02.6486507Z OK (skipped=1) 2022-09-27T16:06:02.6486618Z 2022-09-27T16:06:02.6486704Z Generating XML reports... 2022-09-27T16:06:02.6532268Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160600.xml 2022-09-27T16:06:04.0492729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:04.0493096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:04.0500689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:04.0501278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:04.2080693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9tlvg06c 2022-09-27T16:06:04.2081791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9tlvg06c/_remote_module_non_scriptable.py 2022-09-27T16:06:04.4650726Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:04.4660573Z 2022-09-27T16:06:04.4660684Z Running tests... 2022-09-27T16:06:04.4661113Z ---------------------------------------------------------------------- 2022-09-27T16:06:04.7101902Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3265 2022-09-27T16:06:04.7128626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3266 2022-09-27T16:06:04.7155961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3267 2022-09-27T16:06:04.7184123Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3268 2022-09-27T16:06:05.8266793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:05.8267484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:05.8522090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:05.8522782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:05.8538968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:05.8539725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:05.8623664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:05.8624311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:05.8729468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:05.8730217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:05.8921681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:05.8922239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:05.8935488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:05.8936247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:05.9119550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:05.9120214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:05.9892091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynoecm22 2022-09-27T16:06:05.9893202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynoecm22/_remote_module_non_scriptable.py 2022-09-27T16:06:06.0210667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_btfavd 2022-09-27T16:06:06.0211353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6bg248k 2022-09-27T16:06:06.0212037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_btfavd/_remote_module_non_scriptable.py 2022-09-27T16:06:06.0212618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6bg248k/_remote_module_non_scriptable.py 2022-09-27T16:06:06.0377447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaqppuze_ 2022-09-27T16:06:06.0378408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaqppuze_/_remote_module_non_scriptable.py 2022-09-27T16:06:06.2437824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:06.2766556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:06.2818176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:06.2940359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:06.6238489Z skip: Need at least 2 CUDA devices (2.157s) 2022-09-27T16:06:06.6238910Z 2022-09-27T16:06:06.6239344Z ---------------------------------------------------------------------- 2022-09-27T16:06:06.6239590Z Ran 1 test in 2.158s 2022-09-27T16:06:06.6239701Z 2022-09-27T16:06:06.6239760Z OK (skipped=1) 2022-09-27T16:06:06.6239869Z 2022-09-27T16:06:06.6239953Z Generating XML reports... 2022-09-27T16:06:06.6283850Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160604.xml 2022-09-27T16:06:07.9883863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:07.9884270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:07.9891258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:07.9891615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:08.1442085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperp7s2u1 2022-09-27T16:06:08.1443025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperp7s2u1/_remote_module_non_scriptable.py 2022-09-27T16:06:08.4016561Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:08.4026925Z 2022-09-27T16:06:08.4027344Z Running tests... 2022-09-27T16:06:08.4027751Z ---------------------------------------------------------------------- 2022-09-27T16:06:08.6414302Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3320 2022-09-27T16:06:08.6441680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3321 2022-09-27T16:06:08.6469592Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3322 2022-09-27T16:06:08.6498851Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3323 2022-09-27T16:06:09.7306738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:09.7307373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:09.7465094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:09.7465790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:09.7658886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:09.7659548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:09.7683872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:09.7684551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:09.7851033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:09.7851669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:09.7940399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:09.7941076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:09.8031540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:09.8032205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:09.8299972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:09.8300610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:09.8950107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpguuzksyq 2022-09-27T16:06:09.8950988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpguuzksyq/_remote_module_non_scriptable.py 2022-09-27T16:06:09.9111636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdywy34zh 2022-09-27T16:06:09.9112410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdywy34zh/_remote_module_non_scriptable.py 2022-09-27T16:06:09.9286475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlyb33e5 2022-09-27T16:06:09.9287295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlyb33e5/_remote_module_non_scriptable.py 2022-09-27T16:06:09.9569809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpesn2632r 2022-09-27T16:06:09.9571881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpesn2632r/_remote_module_non_scriptable.py 2022-09-27T16:06:10.1526956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:10.1698689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:10.2147502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:10.2349579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:10.5553248Z skip: Need at least 2 CUDA devices (2.152s) 2022-09-27T16:06:10.5553506Z 2022-09-27T16:06:10.5553895Z ---------------------------------------------------------------------- 2022-09-27T16:06:10.5554232Z Ran 1 test in 2.153s 2022-09-27T16:06:10.5554346Z 2022-09-27T16:06:10.5554406Z OK (skipped=1) 2022-09-27T16:06:10.5554654Z 2022-09-27T16:06:10.5554797Z Generating XML reports... 2022-09-27T16:06:10.5598487Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160608.xml 2022-09-27T16:06:11.9355073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:11.9355486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:11.9364018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:11.9364377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:12.0981064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_gb_tma 2022-09-27T16:06:12.0982058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_gb_tma/_remote_module_non_scriptable.py 2022-09-27T16:06:12.3571580Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:12.3581915Z 2022-09-27T16:06:12.3582304Z Running tests... 2022-09-27T16:06:12.3582704Z ---------------------------------------------------------------------- 2022-09-27T16:06:12.6028908Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3375 2022-09-27T16:06:12.6055966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3376 2022-09-27T16:06:12.6082931Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3377 2022-09-27T16:06:12.6111407Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3378 2022-09-27T16:06:13.7779077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:13.7780052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:13.8145704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:13.8146360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:13.8147191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:13.8147856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:13.8202454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:13.8202953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:13.8512560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:13.8513332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:13.8571361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:13.8571947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:13.8878808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:13.8879451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:13.9230593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:13.9231320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:13.9420215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ddam8r_ 2022-09-27T16:06:13.9421138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ddam8r_/_remote_module_non_scriptable.py 2022-09-27T16:06:13.9773486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplj4ck9ps 2022-09-27T16:06:13.9774755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplj4ck9ps/_remote_module_non_scriptable.py 2022-09-27T16:06:13.9831384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdu37u95 2022-09-27T16:06:13.9833176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdu37u95/_remote_module_non_scriptable.py 2022-09-27T16:06:14.0484862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4ru634z 2022-09-27T16:06:14.0485996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4ru634z/_remote_module_non_scriptable.py 2022-09-27T16:06:14.2015232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:14.2352772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:14.2404521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:14.3142681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:14.6166900Z skip: Need at least 2 CUDA devices (2.258s) 2022-09-27T16:06:14.6167199Z 2022-09-27T16:06:14.6167686Z ---------------------------------------------------------------------- 2022-09-27T16:06:14.6167922Z Ran 1 test in 2.258s 2022-09-27T16:06:14.6168040Z 2022-09-27T16:06:14.6168111Z OK (skipped=1) 2022-09-27T16:06:14.6168220Z 2022-09-27T16:06:14.6168309Z Generating XML reports... 2022-09-27T16:06:14.6212519Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160612.xml 2022-09-27T16:06:15.9741923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:15.9742392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:15.9750674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:15.9751098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:16.1326658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vrotljz 2022-09-27T16:06:16.1327734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vrotljz/_remote_module_non_scriptable.py 2022-09-27T16:06:16.3889364Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:16.3899392Z 2022-09-27T16:06:16.3899476Z Running tests... 2022-09-27T16:06:16.3900430Z ---------------------------------------------------------------------- 2022-09-27T16:06:16.6335829Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3430 2022-09-27T16:06:16.6362359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3431 2022-09-27T16:06:16.6390985Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3432 2022-09-27T16:06:16.6419657Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3433 2022-09-27T16:06:17.7464010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:17.7464619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:17.7586922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:17.7587581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:17.7604717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:17.7605327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:17.7819318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:17.7819974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:17.7831447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:17.7831979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:17.7961525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:17.7962241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:17.7987932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:17.7988521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:17.8168404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:17.8169009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:17.9095874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpask8mr4x 2022-09-27T16:06:17.9096696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpask8mr4x/_remote_module_non_scriptable.py 2022-09-27T16:06:17.9225687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpypf4lhh_ 2022-09-27T16:06:17.9227116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpypf4lhh_/_remote_module_non_scriptable.py 2022-09-27T16:06:17.9258624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm2lzb25j 2022-09-27T16:06:17.9260408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm2lzb25j/_remote_module_non_scriptable.py 2022-09-27T16:06:17.9427193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppm7mcdaz 2022-09-27T16:06:17.9428425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppm7mcdaz/_remote_module_non_scriptable.py 2022-09-27T16:06:18.1652219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:18.1787888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:18.1854975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:18.1969233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:18.5474205Z skip: Need at least 2 CUDA devices (2.157s) 2022-09-27T16:06:18.5474475Z 2022-09-27T16:06:18.5475019Z ---------------------------------------------------------------------- 2022-09-27T16:06:18.5475257Z Ran 1 test in 2.157s 2022-09-27T16:06:18.5475382Z 2022-09-27T16:06:18.5475463Z OK (skipped=1) 2022-09-27T16:06:18.5475573Z 2022-09-27T16:06:18.5475661Z Generating XML reports... 2022-09-27T16:06:18.5521005Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160616.xml 2022-09-27T16:06:19.9325693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:19.9326357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:19.9334840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:19.9335489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:20.0888116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmejyf_n 2022-09-27T16:06:20.0888802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmejyf_n/_remote_module_non_scriptable.py 2022-09-27T16:06:20.3482465Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:20.3492722Z 2022-09-27T16:06:20.3493232Z Running tests... 2022-09-27T16:06:20.3493860Z ---------------------------------------------------------------------- 2022-09-27T16:06:20.5938459Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3485 2022-09-27T16:06:20.5965616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3486 2022-09-27T16:06:20.5992795Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3487 2022-09-27T16:06:20.6022451Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3488 2022-09-27T16:06:21.7617224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:21.7617950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:21.7846511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:21.7847034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:21.7991734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:21.7992476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:21.8014471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:21.8015094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:21.8270062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:21.8270854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:21.8362285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:21.8362960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:21.8377920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:21.8378589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:21.8759064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:21.8759721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:21.9285964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9uc548f 2022-09-27T16:06:21.9286833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9uc548f/_remote_module_non_scriptable.py 2022-09-27T16:06:21.9545088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcve0urr 2022-09-27T16:06:21.9545910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcve0urr/_remote_module_non_scriptable.py 2022-09-27T16:06:21.9634006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2y00vzeq 2022-09-27T16:06:21.9635241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2y00vzeq/_remote_module_non_scriptable.py 2022-09-27T16:06:22.0035602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43frkqe5 2022-09-27T16:06:22.0036348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43frkqe5/_remote_module_non_scriptable.py 2022-09-27T16:06:22.1895505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:22.2142342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:22.2222267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:22.2789571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:22.6077037Z skip: Need at least 2 CUDA devices (2.258s) 2022-09-27T16:06:22.6129268Z 2022-09-27T16:06:22.6129913Z ---------------------------------------------------------------------- 2022-09-27T16:06:22.6130877Z Ran 1 test in 2.258s 2022-09-27T16:06:22.6131121Z 2022-09-27T16:06:22.6131206Z OK (skipped=1) 2022-09-27T16:06:22.6131322Z 2022-09-27T16:06:22.6131414Z Generating XML reports... 2022-09-27T16:06:22.6132098Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160620.xml 2022-09-27T16:06:23.9865989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:23.9866372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:23.9876275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:23.9876632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:24.1512028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx567yrgp 2022-09-27T16:06:24.1512846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx567yrgp/_remote_module_non_scriptable.py 2022-09-27T16:06:24.4069821Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:24.4080441Z 2022-09-27T16:06:24.4080562Z Running tests... 2022-09-27T16:06:24.4081145Z ---------------------------------------------------------------------- 2022-09-27T16:06:24.6515527Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3540 2022-09-27T16:06:24.6541884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3541 2022-09-27T16:06:24.6569550Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3542 2022-09-27T16:06:24.6597965Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3543 2022-09-27T16:06:25.8060501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:25.8061195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:25.8416180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:25.8416853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:25.8439700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:25.8440313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:25.8574098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:25.8574637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:25.8703298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:25.8703797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:25.8838087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:25.8838706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:25.8977546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:25.8978190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:25.9090724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:25.9091396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:25.9711121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9j569wz 2022-09-27T16:06:25.9712525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9j569wz/_remote_module_non_scriptable.py 2022-09-27T16:06:26.0137113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8uws324 2022-09-27T16:06:26.0137928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8uws324/_remote_module_non_scriptable.py 2022-09-27T16:06:26.0368216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhdp6b6d 2022-09-27T16:06:26.0369141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhdp6b6d/_remote_module_non_scriptable.py 2022-09-27T16:06:26.0389660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1_0f23u 2022-09-27T16:06:26.0391202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1_0f23u/_remote_module_non_scriptable.py 2022-09-27T16:06:26.2287352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:26.2691372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:26.2949492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:26.2955742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:26.6656097Z skip: Need at least 2 CUDA devices (2.257s) 2022-09-27T16:06:26.6656288Z 2022-09-27T16:06:26.6656599Z ---------------------------------------------------------------------- 2022-09-27T16:06:26.6657083Z Ran 1 test in 2.257s 2022-09-27T16:06:26.6657203Z 2022-09-27T16:06:26.6657281Z OK (skipped=1) 2022-09-27T16:06:26.6657464Z 2022-09-27T16:06:26.6657552Z Generating XML reports... 2022-09-27T16:06:26.6701872Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160624.xml 2022-09-27T16:06:28.0592176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:28.0592563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:28.0601090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:28.0601504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:28.2147859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8t8ef5o 2022-09-27T16:06:28.2148888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8t8ef5o/_remote_module_non_scriptable.py 2022-09-27T16:06:28.4712375Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:28.4723046Z 2022-09-27T16:06:28.4723180Z Running tests... 2022-09-27T16:06:28.4723768Z ---------------------------------------------------------------------- 2022-09-27T16:06:28.7174628Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3595 2022-09-27T16:06:28.7201254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3596 2022-09-27T16:06:28.7228644Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3597 2022-09-27T16:06:28.7257653Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3598 2022-09-27T16:06:29.8674405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:29.8675218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:29.8706576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:29.8707244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:29.8788551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:29.8789048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:29.8918089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:29.8918595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:29.9065587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:29.9066336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:29.9098525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:29.9099178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:29.9169015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:29.9169637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:29.9281142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:29.9281798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:30.0337120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1su2rf75 2022-09-27T16:06:30.0338091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1su2rf75/_remote_module_non_scriptable.py 2022-09-27T16:06:30.0352183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5d8gzc27 2022-09-27T16:06:30.0354000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5d8gzc27/_remote_module_non_scriptable.py 2022-09-27T16:06:30.0479156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzkh85kh 2022-09-27T16:06:30.0481119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzkh85kh/_remote_module_non_scriptable.py 2022-09-27T16:06:30.0529544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvy6gqqr6 2022-09-27T16:06:30.0531012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvy6gqqr6/_remote_module_non_scriptable.py 2022-09-27T16:06:30.2936163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:30.2942930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:30.3098353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:30.3153733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:30.6312502Z skip: Need at least 2 CUDA devices (2.159s) 2022-09-27T16:06:30.6312795Z 2022-09-27T16:06:30.6313128Z ---------------------------------------------------------------------- 2022-09-27T16:06:30.6313394Z Ran 1 test in 2.159s 2022-09-27T16:06:30.6313510Z 2022-09-27T16:06:30.6313585Z OK (skipped=1) 2022-09-27T16:06:30.6313691Z 2022-09-27T16:06:30.6313778Z Generating XML reports... 2022-09-27T16:06:30.6360247Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160628.xml 2022-09-27T16:06:32.0013399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:32.0013797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:32.0022968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:32.0023328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:32.1597100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcizq7bws 2022-09-27T16:06:32.1598284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcizq7bws/_remote_module_non_scriptable.py 2022-09-27T16:06:32.4185414Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:32.4195473Z 2022-09-27T16:06:32.4195573Z Running tests... 2022-09-27T16:06:32.4196595Z ---------------------------------------------------------------------- 2022-09-27T16:06:32.6619325Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3650 2022-09-27T16:06:32.6646068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3651 2022-09-27T16:06:32.6673503Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3652 2022-09-27T16:06:32.6702766Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3653 2022-09-27T16:06:33.7664323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:33.7664989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:33.7829900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:33.7830421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:33.7831115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:33.7831902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:33.7891563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:33.7892303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:33.8050472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:33.8051075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:33.8218248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:33.8218893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:33.8222039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:33.8222676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:33.8273763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:33.8274410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:33.9314147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgg9jcfl 2022-09-27T16:06:33.9315364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgg9jcfl/_remote_module_non_scriptable.py 2022-09-27T16:06:33.9497235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzi2uhxv 2022-09-27T16:06:33.9497962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeb67wn07 2022-09-27T16:06:33.9498616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzi2uhxv/_remote_module_non_scriptable.py 2022-09-27T16:06:33.9499319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeb67wn07/_remote_module_non_scriptable.py 2022-09-27T16:06:33.9534468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0f5s9ip7 2022-09-27T16:06:33.9536259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0f5s9ip7/_remote_module_non_scriptable.py 2022-09-27T16:06:34.1866907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:34.2079561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:34.2080493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:34.2243542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:34.5757752Z skip: Need at least 1 CUDA device (2.156s) 2022-09-27T16:06:34.5757991Z 2022-09-27T16:06:34.5758423Z ---------------------------------------------------------------------- 2022-09-27T16:06:34.5758774Z Ran 1 test in 2.156s 2022-09-27T16:06:34.5758963Z 2022-09-27T16:06:34.5759051Z OK (skipped=1) 2022-09-27T16:06:34.5759206Z 2022-09-27T16:06:34.5759331Z Generating XML reports... 2022-09-27T16:06:34.5804444Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160632.xml 2022-09-27T16:06:35.9317545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:35.9317912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:35.9325226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:35.9325673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:36.0886367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9w5a3wpg 2022-09-27T16:06:36.0887470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9w5a3wpg/_remote_module_non_scriptable.py 2022-09-27T16:06:36.3500242Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:36.3510671Z 2022-09-27T16:06:36.3510871Z Running tests... 2022-09-27T16:06:36.3512065Z ---------------------------------------------------------------------- 2022-09-27T16:06:36.5997805Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3705 2022-09-27T16:06:36.6024431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3706 2022-09-27T16:06:36.6051747Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3707 2022-09-27T16:06:36.6080179Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3708 2022-09-27T16:06:37.6883125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:37.6883697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:37.6943498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:37.6944130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:37.7147885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:37.7148576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:37.7272870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:37.7273401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:37.7293162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:37.7293653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:37.7331122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:37.7331747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:37.7526285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:37.7526813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:37.7648592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:37.7649251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:37.8535977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvy8iu07 2022-09-27T16:06:37.8536755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvy8iu07/_remote_module_non_scriptable.py 2022-09-27T16:06:37.8580044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiny7thla 2022-09-27T16:06:37.8581584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiny7thla/_remote_module_non_scriptable.py 2022-09-27T16:06:37.8789544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaj91vhbc 2022-09-27T16:06:37.8790222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaj91vhbc/_remote_module_non_scriptable.py 2022-09-27T16:06:37.8901442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ouuwx2s 2022-09-27T16:06:37.8902558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ouuwx2s/_remote_module_non_scriptable.py 2022-09-27T16:06:38.1078206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:38.1126127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:38.1374366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:38.1444468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:38.5134298Z skip: Need at least 2 CUDA devices (2.162s) 2022-09-27T16:06:38.5134496Z 2022-09-27T16:06:38.5134847Z ---------------------------------------------------------------------- 2022-09-27T16:06:38.5135097Z Ran 1 test in 2.162s 2022-09-27T16:06:38.5135211Z 2022-09-27T16:06:38.5135284Z OK (skipped=1) 2022-09-27T16:06:38.5135431Z 2022-09-27T16:06:38.5135527Z Generating XML reports... 2022-09-27T16:06:38.5178739Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160636.xml 2022-09-27T16:06:39.8727988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:39.8728376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:39.8735991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:39.8736492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:40.0341175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0msavi2c 2022-09-27T16:06:40.0342012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0msavi2c/_remote_module_non_scriptable.py 2022-09-27T16:06:40.2910356Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:40.2920568Z 2022-09-27T16:06:40.2920719Z Running tests... 2022-09-27T16:06:40.2921326Z ---------------------------------------------------------------------- 2022-09-27T16:06:40.5347225Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3760 2022-09-27T16:06:40.5375379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3761 2022-09-27T16:06:40.5403918Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3762 2022-09-27T16:06:40.5432305Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3763 2022-09-27T16:06:41.6545910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:41.6546595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:41.6841419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:41.6842025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:41.6898891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:41.6899612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:41.6912038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:41.6912716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:41.6954115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:41.6954844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:41.7250303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:41.7250966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:41.7303384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:41.7304049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:41.7320936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:41.7321665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:41.8220962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfadevl4t 2022-09-27T16:06:41.8221672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfadevl4t/_remote_module_non_scriptable.py 2022-09-27T16:06:41.8514249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgr33r69z 2022-09-27T16:06:41.8515344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgr33r69z/_remote_module_non_scriptable.py 2022-09-27T16:06:41.8564339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9q7xp2v 2022-09-27T16:06:41.8565677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9q7xp2v/_remote_module_non_scriptable.py 2022-09-27T16:06:41.8581651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xxt1y6x 2022-09-27T16:06:41.8583506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xxt1y6x/_remote_module_non_scriptable.py 2022-09-27T16:06:42.0825948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:42.1140752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:42.1172268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:42.1209907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:42.4486775Z skip: Need at least 2 CUDA devices (2.156s) 2022-09-27T16:06:42.4487016Z 2022-09-27T16:06:42.4487326Z ---------------------------------------------------------------------- 2022-09-27T16:06:42.4487564Z Ran 1 test in 2.157s 2022-09-27T16:06:42.4487680Z 2022-09-27T16:06:42.4487752Z OK (skipped=1) 2022-09-27T16:06:42.4487882Z 2022-09-27T16:06:42.4487968Z Generating XML reports... 2022-09-27T16:06:42.4532753Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160640.xml 2022-09-27T16:06:43.8463430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:43.8463817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:43.8472137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:43.8472494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:44.0041845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpljyl489o 2022-09-27T16:06:44.0042794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpljyl489o/_remote_module_non_scriptable.py 2022-09-27T16:06:44.2608990Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:44.2619544Z 2022-09-27T16:06:44.2620095Z Running tests... 2022-09-27T16:06:44.2620636Z ---------------------------------------------------------------------- 2022-09-27T16:06:44.5024228Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3815 2022-09-27T16:06:44.5051246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3816 2022-09-27T16:06:44.5078418Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3817 2022-09-27T16:06:44.5107299Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3818 2022-09-27T16:06:45.6647563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:45.6648505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:45.6931610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:45.6932560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:45.6980442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:45.7001472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:45.7002378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:45.7003045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:45.7021731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:45.7022386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:45.7309192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:45.7309963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:45.7349747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:45.7350375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:45.7392096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:45.7392745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:45.8264663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxydw27t 2022-09-27T16:06:45.8265491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxydw27t/_remote_module_non_scriptable.py 2022-09-27T16:06:45.8572295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdyaa586 2022-09-27T16:06:45.8573231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdyaa586/_remote_module_non_scriptable.py 2022-09-27T16:06:45.8597037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp190bfirh 2022-09-27T16:06:45.8598643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp190bfirh/_remote_module_non_scriptable.py 2022-09-27T16:06:45.8636872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51yqm7r6 2022-09-27T16:06:45.8638420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51yqm7r6/_remote_module_non_scriptable.py 2022-09-27T16:06:46.0819945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:46.1164000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:46.1176759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:46.1211520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:46.4160885Z skip: Need at least 2 CUDA devices (2.154s) 2022-09-27T16:06:46.4161191Z 2022-09-27T16:06:46.4161632Z ---------------------------------------------------------------------- 2022-09-27T16:06:46.4161880Z Ran 1 test in 2.154s 2022-09-27T16:06:46.4161994Z 2022-09-27T16:06:46.4162052Z OK (skipped=1) 2022-09-27T16:06:46.4162161Z 2022-09-27T16:06:46.4162247Z Generating XML reports... 2022-09-27T16:06:46.4206220Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160644.xml 2022-09-27T16:06:47.7785438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:47.7786344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:47.7795038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:47.7795425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:47.9362058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcro4x7z 2022-09-27T16:06:47.9362987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcro4x7z/_remote_module_non_scriptable.py 2022-09-27T16:06:48.1914354Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:48.1924772Z 2022-09-27T16:06:48.1924902Z Running tests... 2022-09-27T16:06:48.1925491Z ---------------------------------------------------------------------- 2022-09-27T16:06:48.4324252Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3870 2022-09-27T16:06:48.4350255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3871 2022-09-27T16:06:48.4377705Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3872 2022-09-27T16:06:48.4406225Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3873 2022-09-27T16:06:49.5477126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:49.5477765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:49.5607412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:49.5608039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:49.5743933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:49.5744587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:49.5850354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:49.5851020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:49.5981764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:49.5982305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:49.6051240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:49.6051851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:49.6117485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:49.6118135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:49.6419325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:49.6419954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:49.7256220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47l3mljj 2022-09-27T16:06:49.7257045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47l3mljj/_remote_module_non_scriptable.py 2022-09-27T16:06:49.7308176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3nkah6n0 2022-09-27T16:06:49.7309576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3nkah6n0/_remote_module_non_scriptable.py 2022-09-27T16:06:49.7397384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvwwlifz7 2022-09-27T16:06:49.7398932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvwwlifz7/_remote_module_non_scriptable.py 2022-09-27T16:06:49.7721937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_9uor40 2022-09-27T16:06:49.7723080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_9uor40/_remote_module_non_scriptable.py 2022-09-27T16:06:49.9799986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:49.9850506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:49.9918723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:50.0421896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:50.3460983Z skip: Need at least 1 CUDA device (2.153s) 2022-09-27T16:06:50.3461294Z 2022-09-27T16:06:50.3461705Z ---------------------------------------------------------------------- 2022-09-27T16:06:50.3461969Z Ran 1 test in 2.154s 2022-09-27T16:06:50.3462086Z 2022-09-27T16:06:50.3462146Z OK (skipped=1) 2022-09-27T16:06:50.3462256Z 2022-09-27T16:06:50.3462342Z Generating XML reports... 2022-09-27T16:06:50.3506844Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160648.xml 2022-09-27T16:06:51.7001131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:51.7001544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:51.7009611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:51.7010125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:51.8581080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbn5apafv 2022-09-27T16:06:51.8582113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbn5apafv/_remote_module_non_scriptable.py 2022-09-27T16:06:52.1184215Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:52.1194493Z 2022-09-27T16:06:52.1194843Z Running tests... 2022-09-27T16:06:52.1195437Z ---------------------------------------------------------------------- 2022-09-27T16:06:52.3629795Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3925 2022-09-27T16:06:52.3657333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3926 2022-09-27T16:06:52.3685301Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3927 2022-09-27T16:06:52.3713794Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3928 2022-09-27T16:06:53.5294065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:53.5294708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:53.5654308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:53.5654912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:53.5658200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:53.5658900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:53.5851261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:53.5851797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:53.5936795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:53.5937465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:53.6039962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:53.6040602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:53.6231368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:53.6232165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:53.6320561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:53.6321195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:53.6934443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__c5mbi1 2022-09-27T16:06:53.6935448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__c5mbi1/_remote_module_non_scriptable.py 2022-09-27T16:06:53.7303432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpix6sp__i 2022-09-27T16:06:53.7304172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpix6sp__i/_remote_module_non_scriptable.py 2022-09-27T16:06:53.7499985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8xiwjlk2 2022-09-27T16:06:53.7500995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8xiwjlk2/_remote_module_non_scriptable.py 2022-09-27T16:06:53.7585369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3tb35lx 2022-09-27T16:06:53.7586102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3tb35lx/_remote_module_non_scriptable.py 2022-09-27T16:06:53.9507799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:54.0064272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:54.0091829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:54.0177393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:54.3770663Z skip: Need at least 1 CUDA device (2.257s) 2022-09-27T16:06:54.3770918Z 2022-09-27T16:06:54.3771230Z ---------------------------------------------------------------------- 2022-09-27T16:06:54.3771548Z Ran 1 test in 2.258s 2022-09-27T16:06:54.3771652Z 2022-09-27T16:06:54.3771726Z OK (skipped=1) 2022-09-27T16:06:54.3771838Z 2022-09-27T16:06:54.3771923Z Generating XML reports... 2022-09-27T16:06:54.3816915Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160652.xml 2022-09-27T16:06:55.7607788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:55.7608167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:55.7616279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:55.7616646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:55.9304365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7wtp67d 2022-09-27T16:06:55.9305658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7wtp67d/_remote_module_non_scriptable.py 2022-09-27T16:06:56.1877119Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:06:56.1887702Z 2022-09-27T16:06:56.1888174Z Running tests... 2022-09-27T16:06:56.1888594Z ---------------------------------------------------------------------- 2022-09-27T16:06:56.4320752Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3980 2022-09-27T16:06:56.4348279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3981 2022-09-27T16:06:56.4376651Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3982 2022-09-27T16:06:56.4405614Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3983 2022-09-27T16:06:57.5248690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:57.5249369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:57.5257078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:57.5257707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:57.5622787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:57.5623504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:57.5627868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:57.5628908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:57.5629793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:57.5630485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:57.5639787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:57.5640310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:57.5992038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:57.5992655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:57.5995928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:57.5996543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:57.6902845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgg8sscmw 2022-09-27T16:06:57.6903525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplisvg2fr 2022-09-27T16:06:57.6904187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgg8sscmw/_remote_module_non_scriptable.py 2022-09-27T16:06:57.6904804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplisvg2fr/_remote_module_non_scriptable.py 2022-09-27T16:06:57.7258823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjnotg55 2022-09-27T16:06:57.7259590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjnotg55/_remote_module_non_scriptable.py 2022-09-27T16:06:57.7271146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzs5yjy55 2022-09-27T16:06:57.7273285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzs5yjy55/_remote_module_non_scriptable.py 2022-09-27T16:06:57.9455949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:06:57.9461858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:06:57.9849148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:06:57.9939827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:06:58.3460112Z skip: Need at least 1 CUDA device (2.157s) 2022-09-27T16:06:58.3460470Z 2022-09-27T16:06:58.3460966Z ---------------------------------------------------------------------- 2022-09-27T16:06:58.3461226Z Ran 1 test in 2.157s 2022-09-27T16:06:58.3461343Z 2022-09-27T16:06:58.3461417Z OK (skipped=1) 2022-09-27T16:06:58.3461514Z 2022-09-27T16:06:58.3461602Z Generating XML reports... 2022-09-27T16:06:58.3506588Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160656.xml 2022-09-27T16:06:59.7104422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:06:59.7104805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:06:59.7112134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:06:59.7112479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:06:59.8685369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnxq9toa 2022-09-27T16:06:59.8686419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnxq9toa/_remote_module_non_scriptable.py 2022-09-27T16:07:00.1258626Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:00.1269179Z 2022-09-27T16:07:00.1269280Z Running tests... 2022-09-27T16:07:00.1269715Z ---------------------------------------------------------------------- 2022-09-27T16:07:00.3692437Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4035 2022-09-27T16:07:00.3719245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4036 2022-09-27T16:07:00.3746199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4037 2022-09-27T16:07:00.3775179Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4038 2022-09-27T16:07:01.5648043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:01.5648694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:01.5702073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:01.5702738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:01.5927644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:01.5928156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:01.6035580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:01.6036317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:01.6106594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:01.6107206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:01.6202713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:01.6203342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:01.6318484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:01.6319157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:01.6592010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:01.6592587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:01.7322839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7nlcpddp 2022-09-27T16:07:01.7323624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7nlcpddp/_remote_module_non_scriptable.py 2022-09-27T16:07:01.7399862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51uol0o7 2022-09-27T16:07:01.7400937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51uol0o7/_remote_module_non_scriptable.py 2022-09-27T16:07:01.7596708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp176ck5g 2022-09-27T16:07:01.7597814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp176ck5g/_remote_module_non_scriptable.py 2022-09-27T16:07:01.7862106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4xva4nj 2022-09-27T16:07:01.7862909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4xva4nj/_remote_module_non_scriptable.py 2022-09-27T16:07:01.9964826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:02.0076946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:02.0263838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:02.0709532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:02.3832206Z skip: Need at least 2 CUDA devices (2.256s) 2022-09-27T16:07:02.3832659Z 2022-09-27T16:07:02.3833122Z ---------------------------------------------------------------------- 2022-09-27T16:07:02.3833511Z Ran 1 test in 2.256s 2022-09-27T16:07:02.3833682Z 2022-09-27T16:07:02.3833798Z OK (skipped=1) 2022-09-27T16:07:02.3833961Z 2022-09-27T16:07:02.3834099Z Generating XML reports... 2022-09-27T16:07:02.3879964Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160700.xml 2022-09-27T16:07:03.7801321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:03.7801695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:03.7810069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:03.7810446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:03.9365616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0jxquk9a 2022-09-27T16:07:03.9366851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0jxquk9a/_remote_module_non_scriptable.py 2022-09-27T16:07:04.1932441Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:04.1943379Z 2022-09-27T16:07:04.1943499Z Running tests... 2022-09-27T16:07:04.1944481Z ---------------------------------------------------------------------- 2022-09-27T16:07:04.4344252Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4090 2022-09-27T16:07:04.4372614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4091 2022-09-27T16:07:04.4402194Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4092 2022-09-27T16:07:04.4431293Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4093 2022-09-27T16:07:05.6146677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:05.6147359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:05.6265917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:05.6266548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:05.6524147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:05.6524838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:05.6545660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:05.6546637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:05.6660481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:05.6661549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:05.6890061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:05.6890709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:05.6905268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:05.6905881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:05.7274706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:05.7275440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:05.7828486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppc83sxge 2022-09-27T16:07:05.7829570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppc83sxge/_remote_module_non_scriptable.py 2022-09-27T16:07:05.7942148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1cxbdex 2022-09-27T16:07:05.7942922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1cxbdex/_remote_module_non_scriptable.py 2022-09-27T16:07:05.8184597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1w9_kii 2022-09-27T16:07:05.8185330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1w9_kii/_remote_module_non_scriptable.py 2022-09-27T16:07:05.8542372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp883jf84e 2022-09-27T16:07:05.8543209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp883jf84e/_remote_module_non_scriptable.py 2022-09-27T16:07:06.0442298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:06.0546161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:06.0788947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:06.1283750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:06.4486383Z skip: Need at least 1 CUDA device (2.254s) 2022-09-27T16:07:06.4486686Z 2022-09-27T16:07:06.4487186Z ---------------------------------------------------------------------- 2022-09-27T16:07:06.4487603Z Ran 1 test in 2.254s 2022-09-27T16:07:06.4487717Z 2022-09-27T16:07:06.4487780Z OK (skipped=1) 2022-09-27T16:07:06.4487893Z 2022-09-27T16:07:06.4487981Z Generating XML reports... 2022-09-27T16:07:06.4533864Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160704.xml 2022-09-27T16:07:07.8653097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:07.8653479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:07.8661789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:07.8662148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:08.0253188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9t_7ub5b 2022-09-27T16:07:08.0254573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9t_7ub5b/_remote_module_non_scriptable.py 2022-09-27T16:07:08.2863753Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:08.2874136Z 2022-09-27T16:07:08.2874337Z Running tests... 2022-09-27T16:07:08.2875323Z ---------------------------------------------------------------------- 2022-09-27T16:07:08.5342557Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4145 2022-09-27T16:07:08.5369758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4146 2022-09-27T16:07:08.5396983Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4147 2022-09-27T16:07:08.5427173Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4148 2022-09-27T16:07:09.7327816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:09.7328495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:09.7351230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:09.7351870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:09.7622412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:09.7623148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:09.7713766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:09.7714297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:09.7741487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:09.7742372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:09.7844286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:09.7844915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:09.8000967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:09.8001654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:09.8212034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:09.8212694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:09.8989665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_091g4q 2022-09-27T16:07:09.8990463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_091g4q/_remote_module_non_scriptable.py 2022-09-27T16:07:09.9019313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6141q68d 2022-09-27T16:07:09.9020505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6141q68d/_remote_module_non_scriptable.py 2022-09-27T16:07:09.9264234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikpyqarl 2022-09-27T16:07:09.9265027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikpyqarl/_remote_module_non_scriptable.py 2022-09-27T16:07:09.9465497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5v617f5 2022-09-27T16:07:09.9466946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5v617f5/_remote_module_non_scriptable.py 2022-09-27T16:07:10.1550442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:10.1614850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:10.1823876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:10.2111052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:10.5483506Z skip: Need at least 1 CUDA device (2.261s) 2022-09-27T16:07:10.5483818Z 2022-09-27T16:07:10.5484307Z ---------------------------------------------------------------------- 2022-09-27T16:07:10.5484626Z Ran 1 test in 2.261s 2022-09-27T16:07:10.5484739Z 2022-09-27T16:07:10.5484813Z OK (skipped=1) 2022-09-27T16:07:10.5484923Z 2022-09-27T16:07:10.5485009Z Generating XML reports... 2022-09-27T16:07:10.5528473Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160708.xml 2022-09-27T16:07:11.9256184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:11.9256552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:11.9265017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:11.9265403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:12.0869672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3cmd4bn4 2022-09-27T16:07:12.0870838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3cmd4bn4/_remote_module_non_scriptable.py 2022-09-27T16:07:12.3458814Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:12.3469664Z 2022-09-27T16:07:12.3469757Z Running tests... 2022-09-27T16:07:12.3470557Z ---------------------------------------------------------------------- 2022-09-27T16:07:12.5948605Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4200 2022-09-27T16:07:12.5975679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4201 2022-09-27T16:07:12.6003558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4202 2022-09-27T16:07:12.6033233Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4203 2022-09-27T16:07:13.6934803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:13.6948637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:13.6981918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:13.6982537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:13.7070946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:13.7071462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:13.7191272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:13.7191772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:13.7314140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:13.7314935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:13.7366900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:13.7367480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:13.7456926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:13.7457557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:13.7563351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:13.7563977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:13.8575356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3kbm2_w 2022-09-27T16:07:13.8576215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3kbm2_w/_remote_module_non_scriptable.py 2022-09-27T16:07:13.8626012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4sxq8w2 2022-09-27T16:07:13.8627532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4sxq8w2/_remote_module_non_scriptable.py 2022-09-27T16:07:13.8714006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdamprdzp 2022-09-27T16:07:13.8715266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdamprdzp/_remote_module_non_scriptable.py 2022-09-27T16:07:13.8816149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpenk420s0 2022-09-27T16:07:13.8818039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpenk420s0/_remote_module_non_scriptable.py 2022-09-27T16:07:14.1128964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:14.1178282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:14.1289109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:14.1447116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:14.5086470Z skip: Need at least 1 CUDA device (2.161s) 2022-09-27T16:07:14.5086782Z 2022-09-27T16:07:14.5087256Z ---------------------------------------------------------------------- 2022-09-27T16:07:14.5087682Z Ran 1 test in 2.162s 2022-09-27T16:07:14.5087830Z 2022-09-27T16:07:14.5087904Z OK (skipped=1) 2022-09-27T16:07:14.5088014Z 2022-09-27T16:07:14.5088100Z Generating XML reports... 2022-09-27T16:07:14.5133521Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160712.xml 2022-09-27T16:07:15.8942029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:15.8942396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:15.8950306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:15.8950734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:16.0527454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8xs8qza 2022-09-27T16:07:16.0528385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8xs8qza/_remote_module_non_scriptable.py 2022-09-27T16:07:16.3095614Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:16.3106096Z 2022-09-27T16:07:16.3106213Z Running tests... 2022-09-27T16:07:16.3106682Z ---------------------------------------------------------------------- 2022-09-27T16:07:16.5522854Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4255 2022-09-27T16:07:16.5550407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4256 2022-09-27T16:07:16.5577558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4257 2022-09-27T16:07:16.5607173Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4258 2022-09-27T16:07:17.6848143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:17.6848818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:17.7025546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:17.7026227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:17.7228958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:17.7229650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:17.7352879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:17.7353589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:17.7393304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:17.7393952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:17.7427485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:17.7428054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:17.7731743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:17.7732423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:17.7806497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:17.7807151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:17.8500859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9b286mpx 2022-09-27T16:07:17.8501801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9b286mpx/_remote_module_non_scriptable.py 2022-09-27T16:07:17.8663896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwuo92nx8 2022-09-27T16:07:17.8664806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwuo92nx8/_remote_module_non_scriptable.py 2022-09-27T16:07:17.9001644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmr_s15sj 2022-09-27T16:07:17.9002516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmr_s15sj/_remote_module_non_scriptable.py 2022-09-27T16:07:17.9065850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphc9sql_k 2022-09-27T16:07:17.9067318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphc9sql_k/_remote_module_non_scriptable.py 2022-09-27T16:07:18.1085483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:18.1240360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:18.1741573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:18.1783240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:18.4660955Z skip: Need at least 1 CUDA device (2.155s) 2022-09-27T16:07:18.4661149Z 2022-09-27T16:07:18.4661584Z ---------------------------------------------------------------------- 2022-09-27T16:07:18.4662048Z Ran 1 test in 2.155s 2022-09-27T16:07:18.4662267Z 2022-09-27T16:07:18.4662360Z OK (skipped=1) 2022-09-27T16:07:18.4662476Z 2022-09-27T16:07:18.4662566Z Generating XML reports... 2022-09-27T16:07:18.4707424Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160716.xml 2022-09-27T16:07:19.8546347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:19.8546732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:19.8554924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:19.8555304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:20.0125285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1e218eck 2022-09-27T16:07:20.0126362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1e218eck/_remote_module_non_scriptable.py 2022-09-27T16:07:20.2690425Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:20.2700551Z 2022-09-27T16:07:20.2700659Z Running tests... 2022-09-27T16:07:20.2701138Z ---------------------------------------------------------------------- 2022-09-27T16:07:20.5151414Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4310 2022-09-27T16:07:20.5179535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4311 2022-09-27T16:07:20.5206967Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4312 2022-09-27T16:07:20.5236426Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4313 2022-09-27T16:07:21.6461170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:21.6461841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:21.6652218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:21.6652911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:21.6653700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:21.6654327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:21.6854287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:21.6854839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:21.6956408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:21.6957055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:21.7041155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:21.7041776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:21.7042489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:21.7043033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:21.7345169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:21.7345806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:21.8135000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28znnzvf 2022-09-27T16:07:21.8135807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28znnzvf/_remote_module_non_scriptable.py 2022-09-27T16:07:21.8309309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyr314fr 2022-09-27T16:07:21.8310426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyr314fr/_remote_module_non_scriptable.py 2022-09-27T16:07:21.8489796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_rgrw_h 2022-09-27T16:07:21.8490936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_rgrw_h/_remote_module_non_scriptable.py 2022-09-27T16:07:21.8640052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcwliy7hf 2022-09-27T16:07:21.8641549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcwliy7hf/_remote_module_non_scriptable.py 2022-09-27T16:07:22.0693182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:22.0890586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:22.1055142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:22.1288323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:22.4289621Z skip: Need at least 1 CUDA device (2.159s) 2022-09-27T16:07:22.4290166Z 2022-09-27T16:07:22.4290757Z ---------------------------------------------------------------------- 2022-09-27T16:07:22.4291226Z Ran 1 test in 2.159s 2022-09-27T16:07:22.4291441Z 2022-09-27T16:07:22.4291512Z OK (skipped=1) 2022-09-27T16:07:22.4291624Z 2022-09-27T16:07:22.4291713Z Generating XML reports... 2022-09-27T16:07:22.4335053Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160720.xml 2022-09-27T16:07:23.8016565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:23.8016953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:23.8025560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:23.8025919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:23.9617148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9jrsq4a 2022-09-27T16:07:23.9618096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9jrsq4a/_remote_module_non_scriptable.py 2022-09-27T16:07:24.2247148Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:24.2257142Z 2022-09-27T16:07:24.2257282Z Running tests... 2022-09-27T16:07:24.2257857Z ---------------------------------------------------------------------- 2022-09-27T16:07:24.4707764Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4365 2022-09-27T16:07:24.4734884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4366 2022-09-27T16:07:24.4762220Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4367 2022-09-27T16:07:24.4792323Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4368 2022-09-27T16:07:25.6272212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:25.6272823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:25.6345861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:25.6346456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:25.6372026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:25.6372521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:25.6415784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:25.6416447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:25.6660647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:25.6661300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:25.6733325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:25.6734072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:25.6766556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:25.6767196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:25.6802283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:25.6802891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:25.7926311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3epd657 2022-09-27T16:07:25.7927119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3epd657/_remote_module_non_scriptable.py 2022-09-27T16:07:25.7990127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfp_vjz24 2022-09-27T16:07:25.7991382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfp_vjz24/_remote_module_non_scriptable.py 2022-09-27T16:07:25.8035579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplb_290ih 2022-09-27T16:07:25.8037377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplb_290ih/_remote_module_non_scriptable.py 2022-09-27T16:07:25.8057514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_f0ul4c 2022-09-27T16:07:25.8059003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_f0ul4c/_remote_module_non_scriptable.py 2022-09-27T16:07:26.0547069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:26.0612881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:26.0636062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:26.0679639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:26.3845156Z skip: Need at least 1 CUDA device (2.158s) 2022-09-27T16:07:26.3845456Z 2022-09-27T16:07:26.3845818Z ---------------------------------------------------------------------- 2022-09-27T16:07:26.3846077Z Ran 1 test in 2.159s 2022-09-27T16:07:26.3846196Z 2022-09-27T16:07:26.3846264Z OK (skipped=1) 2022-09-27T16:07:26.3846376Z 2022-09-27T16:07:26.3846465Z Generating XML reports... 2022-09-27T16:07:26.3891980Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160724.xml 2022-09-27T16:07:27.7686363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:27.7686746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:27.7694128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:27.7694494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:27.9261025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb9e0jiue 2022-09-27T16:07:27.9262207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb9e0jiue/_remote_module_non_scriptable.py 2022-09-27T16:07:28.1822599Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:28.1832794Z 2022-09-27T16:07:28.1833123Z Running tests... 2022-09-27T16:07:28.1833521Z ---------------------------------------------------------------------- 2022-09-27T16:07:28.4259310Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4420 2022-09-27T16:07:28.4285309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4421 2022-09-27T16:07:28.4311806Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4422 2022-09-27T16:07:28.4340331Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4423 2022-09-27T16:07:29.5890659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:29.5891388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:29.6034986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:29.6035599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:29.6271772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:29.6272413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:29.6287804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:29.6288445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:29.6419214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:29.6419943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:29.6660988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:29.6661643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:29.6944138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:29.6944796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:29.7323089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:29.7323779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:29.7540299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwqizqh_ 2022-09-27T16:07:29.7541074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwqizqh_/_remote_module_non_scriptable.py 2022-09-27T16:07:29.7692476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0j8rfjay 2022-09-27T16:07:29.7693226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0j8rfjay/_remote_module_non_scriptable.py 2022-09-27T16:07:29.7946089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppc7ptl9b 2022-09-27T16:07:29.7946927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppc7ptl9b/_remote_module_non_scriptable.py 2022-09-27T16:07:29.8585867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8pg0nxs 2022-09-27T16:07:29.8586656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8pg0nxs/_remote_module_non_scriptable.py 2022-09-27T16:07:30.0129723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:30.0252382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:30.0526619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:30.1189322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:30.4397473Z skip: Need at least 1 CUDA device (2.256s) 2022-09-27T16:07:30.4397744Z 2022-09-27T16:07:30.4398054Z ---------------------------------------------------------------------- 2022-09-27T16:07:30.4398305Z Ran 1 test in 2.256s 2022-09-27T16:07:30.4398418Z 2022-09-27T16:07:30.4398478Z OK (skipped=1) 2022-09-27T16:07:30.4398585Z 2022-09-27T16:07:30.4398670Z Generating XML reports... 2022-09-27T16:07:30.4443090Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160728.xml 2022-09-27T16:07:31.8352069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:31.8352476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:31.8360408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:31.8360858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:31.9968334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpxw7twh 2022-09-27T16:07:31.9969505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpxw7twh/_remote_module_non_scriptable.py 2022-09-27T16:07:32.2551465Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:32.2561994Z 2022-09-27T16:07:32.2562266Z Running tests... 2022-09-27T16:07:32.2562618Z ---------------------------------------------------------------------- 2022-09-27T16:07:32.5042145Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4475 2022-09-27T16:07:32.5069708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4476 2022-09-27T16:07:32.5098137Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4477 2022-09-27T16:07:32.5127683Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4478 2022-09-27T16:07:33.6482728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:33.6483339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:33.6515587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:33.6516257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:33.6677806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:33.6678326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:33.6859877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:33.6860621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:33.6897823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:33.6898344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:33.7050485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:33.7269526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:33.7270459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:33.7271141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:33.7618821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:33.7619522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:33.8127331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppilx1exz 2022-09-27T16:07:33.8128106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppilx1exz/_remote_module_non_scriptable.py 2022-09-27T16:07:33.8164178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjuq6ole 2022-09-27T16:07:33.8165841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjuq6ole/_remote_module_non_scriptable.py 2022-09-27T16:07:33.8309775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjijl0j_d 2022-09-27T16:07:33.8310947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjijl0j_d/_remote_module_non_scriptable.py 2022-09-27T16:07:33.8874078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvtyalei1 2022-09-27T16:07:33.8875288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvtyalei1/_remote_module_non_scriptable.py 2022-09-27T16:07:34.0695088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:34.0738290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:34.0859768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:34.1448699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:34.4181895Z skip: Need at least 2 CUDA devices (2.162s) 2022-09-27T16:07:34.4182170Z 2022-09-27T16:07:34.4182684Z ---------------------------------------------------------------------- 2022-09-27T16:07:34.4183018Z Ran 1 test in 2.162s 2022-09-27T16:07:34.4183133Z 2022-09-27T16:07:34.4183192Z OK (skipped=1) 2022-09-27T16:07:34.4183301Z 2022-09-27T16:07:34.4183392Z Generating XML reports... 2022-09-27T16:07:34.4228185Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160732.xml 2022-09-27T16:07:35.7934837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:35.7935226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:35.7942725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:35.7943223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:35.9468991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3x6my97 2022-09-27T16:07:35.9470151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3x6my97/_remote_module_non_scriptable.py 2022-09-27T16:07:36.2002951Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:36.2013114Z 2022-09-27T16:07:36.2013252Z Running tests... 2022-09-27T16:07:36.2013952Z ---------------------------------------------------------------------- 2022-09-27T16:07:36.4479982Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4530 2022-09-27T16:07:36.4505933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4531 2022-09-27T16:07:36.4533885Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4532 2022-09-27T16:07:36.4562809Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4533 2022-09-27T16:07:37.5318499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:37.5319172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:37.5367158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:37.5367824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:37.5691744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:37.5692377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:37.5732005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:37.5732657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:37.5790628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:37.5791611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:37.6027597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:37.6028522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:37.6173489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:37.6174110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:37.6399723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:37.6400392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:37.6970840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54yw848w 2022-09-27T16:07:37.6971623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54yw848w/_remote_module_non_scriptable.py 2022-09-27T16:07:37.7005319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6ptykti 2022-09-27T16:07:37.7007150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6ptykti/_remote_module_non_scriptable.py 2022-09-27T16:07:37.7429988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0n_re_oj 2022-09-27T16:07:37.7430849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0n_re_oj/_remote_module_non_scriptable.py 2022-09-27T16:07:37.7665145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpymfoov5k 2022-09-27T16:07:37.7665933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpymfoov5k/_remote_module_non_scriptable.py 2022-09-27T16:07:37.9566610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:37.9583441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:38.0181166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:38.0262518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:38.3628465Z skip: Need at least 2 CUDA devices (2.161s) 2022-09-27T16:07:38.3628780Z 2022-09-27T16:07:38.3629206Z ---------------------------------------------------------------------- 2022-09-27T16:07:38.3629442Z Ran 1 test in 2.161s 2022-09-27T16:07:38.3629555Z 2022-09-27T16:07:38.3629627Z OK (skipped=1) 2022-09-27T16:07:38.3629738Z 2022-09-27T16:07:38.3629822Z Generating XML reports... 2022-09-27T16:07:38.3675313Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160736.xml 2022-09-27T16:07:39.7283013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:39.7283394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:39.7291591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:39.7292119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:39.8849901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb3kf7n3i 2022-09-27T16:07:39.8851029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb3kf7n3i/_remote_module_non_scriptable.py 2022-09-27T16:07:40.1420893Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:40.1431382Z 2022-09-27T16:07:40.1431897Z Running tests... 2022-09-27T16:07:40.1432298Z ---------------------------------------------------------------------- 2022-09-27T16:07:40.3891432Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4585 2022-09-27T16:07:40.3919659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4586 2022-09-27T16:07:40.3948623Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4587 2022-09-27T16:07:40.3978366Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4588 2022-09-27T16:07:41.5043304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:41.5043929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:41.5084442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:41.5085096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:41.5402691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:41.5403456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:41.5424918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:41.5425521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:41.5446059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:41.5446693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:41.5631203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:41.5631851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:41.5808077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:41.5808760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:41.6022725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:41.6023454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:41.6674884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ummke5f 2022-09-27T16:07:41.6675970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ummke5f/_remote_module_non_scriptable.py 2022-09-27T16:07:41.6721344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzycclp0d 2022-09-27T16:07:41.6722469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzycclp0d/_remote_module_non_scriptable.py 2022-09-27T16:07:41.7101843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpni1grzwq 2022-09-27T16:07:41.7102562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpni1grzwq/_remote_module_non_scriptable.py 2022-09-27T16:07:41.7299462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2boztd9 2022-09-27T16:07:41.7300728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2boztd9/_remote_module_non_scriptable.py 2022-09-27T16:07:41.9230595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:41.9285671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:41.9850831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:41.9864617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:42.3032476Z skip: Need at least 2 CUDA devices (2.160s) 2022-09-27T16:07:42.3032734Z 2022-09-27T16:07:42.3033047Z ---------------------------------------------------------------------- 2022-09-27T16:07:42.3033294Z Ran 1 test in 2.160s 2022-09-27T16:07:42.3033456Z 2022-09-27T16:07:42.3033767Z OK (skipped=1) 2022-09-27T16:07:42.3033881Z 2022-09-27T16:07:42.3033954Z Generating XML reports... 2022-09-27T16:07:42.3077981Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160740.xml 2022-09-27T16:07:43.6787808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:43.6788232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:43.6796983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:43.6797346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:43.8381421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbj9hp9_ 2022-09-27T16:07:43.8382903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbj9hp9_/_remote_module_non_scriptable.py 2022-09-27T16:07:44.0947047Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:44.0957594Z 2022-09-27T16:07:44.0957704Z Running tests... 2022-09-27T16:07:44.0958171Z ---------------------------------------------------------------------- 2022-09-27T16:07:44.3350103Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4640 2022-09-27T16:07:44.3377175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4641 2022-09-27T16:07:44.3404217Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4642 2022-09-27T16:07:44.3432708Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4643 2022-09-27T16:07:45.5230220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:45.5230920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:45.5270381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:45.5271137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:45.5598472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:45.5599162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:45.5654919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:45.5655492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:45.5687147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:45.5687665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:45.5740370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:45.5741017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:45.5975593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:45.5976099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:45.6112467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:45.6113101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:45.6929850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp810w0357 2022-09-27T16:07:45.6930641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp810w0357/_remote_module_non_scriptable.py 2022-09-27T16:07:45.7041056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvd7xa75q 2022-09-27T16:07:45.7042304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvd7xa75q/_remote_module_non_scriptable.py 2022-09-27T16:07:45.7232251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcevptjx 2022-09-27T16:07:45.7233576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcevptjx/_remote_module_non_scriptable.py 2022-09-27T16:07:45.7369102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0c7y9fs 2022-09-27T16:07:45.7370119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0c7y9fs/_remote_module_non_scriptable.py 2022-09-27T16:07:45.9499907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:45.9605043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:45.9780869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:45.9936236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:46.3489240Z skip: Need at least 2 CUDA devices (2.253s) 2022-09-27T16:07:46.3489538Z 2022-09-27T16:07:46.3490071Z ---------------------------------------------------------------------- 2022-09-27T16:07:46.3490336Z Ran 1 test in 2.253s 2022-09-27T16:07:46.3490437Z 2022-09-27T16:07:46.3490508Z OK (skipped=1) 2022-09-27T16:07:46.3490614Z 2022-09-27T16:07:46.3490699Z Generating XML reports... 2022-09-27T16:07:46.3537343Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160744.xml 2022-09-27T16:07:47.7073463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:47.7074141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:47.7084506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:47.7085069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:47.8741910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49dcba_3 2022-09-27T16:07:47.8742774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49dcba_3/_remote_module_non_scriptable.py 2022-09-27T16:07:48.1287753Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:48.1298789Z 2022-09-27T16:07:48.1299072Z Running tests... 2022-09-27T16:07:48.1299704Z ---------------------------------------------------------------------- 2022-09-27T16:07:48.3704817Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4695 2022-09-27T16:07:48.3732534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4696 2022-09-27T16:07:48.3760069Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4697 2022-09-27T16:07:48.3789582Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4698 2022-09-27T16:07:49.4608757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:49.4609350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:49.4965661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:49.4966347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:49.4990058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:49.4990763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:49.5012506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:49.5013219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:49.5195535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:49.5196064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:49.5380086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:49.5380818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:49.5397953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:49.5398607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:49.5568241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:49.5568836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:49.6229736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejwut_mz 2022-09-27T16:07:49.6230534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejwut_mz/_remote_module_non_scriptable.py 2022-09-27T16:07:49.6649629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25gjlqvl 2022-09-27T16:07:49.6650405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25gjlqvl/_remote_module_non_scriptable.py 2022-09-27T16:07:49.6663476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rh9tble 2022-09-27T16:07:49.6664990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rh9tble/_remote_module_non_scriptable.py 2022-09-27T16:07:49.6832480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9ddj9jx 2022-09-27T16:07:49.6834385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9ddj9jx/_remote_module_non_scriptable.py 2022-09-27T16:07:49.8993525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:49.9209482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:49.9258958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:49.9413894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:50.2844101Z skip: Need at least 4 CUDA devices (2.154s) 2022-09-27T16:07:50.2844372Z 2022-09-27T16:07:50.2844760Z ---------------------------------------------------------------------- 2022-09-27T16:07:50.2845072Z Ran 1 test in 2.154s 2022-09-27T16:07:50.2845186Z 2022-09-27T16:07:50.2845261Z OK (skipped=1) 2022-09-27T16:07:50.2845371Z 2022-09-27T16:07:50.2845475Z Generating XML reports... 2022-09-27T16:07:50.2889337Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160748.xml 2022-09-27T16:07:51.6640525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:51.6640957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:51.6649060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:51.6649419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:51.8246064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryxaexkx 2022-09-27T16:07:51.8246901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryxaexkx/_remote_module_non_scriptable.py 2022-09-27T16:07:52.0878684Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:52.0889015Z 2022-09-27T16:07:52.0889114Z Running tests... 2022-09-27T16:07:52.0889636Z ---------------------------------------------------------------------- 2022-09-27T16:07:52.3343340Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4750 2022-09-27T16:07:52.3370719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4751 2022-09-27T16:07:52.3398427Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4752 2022-09-27T16:07:52.3427046Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4753 2022-09-27T16:07:53.4324216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:53.4324911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:53.4325706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:53.4326258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:53.4608193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:53.4608832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:53.4721053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:53.4721654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:53.4724141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:53.4841727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:53.4842510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:53.4842995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:53.4986014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:53.4986640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:53.5208406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:53.5209056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:53.5997855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwz4wovs 2022-09-27T16:07:53.6000612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1twpfyx 2022-09-27T16:07:53.6001417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwz4wovs/_remote_module_non_scriptable.py 2022-09-27T16:07:53.6002139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1twpfyx/_remote_module_non_scriptable.py 2022-09-27T16:07:53.6255430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26e25pmv 2022-09-27T16:07:53.6256203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26e25pmv/_remote_module_non_scriptable.py 2022-09-27T16:07:53.6469693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfaxu954 2022-09-27T16:07:53.6470827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfaxu954/_remote_module_non_scriptable.py 2022-09-27T16:07:53.8583843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:53.8702123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:53.8852533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:53.9186476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:54.2482775Z skip: Need at least 4 CUDA devices (2.159s) 2022-09-27T16:07:54.2483482Z 2022-09-27T16:07:54.2483788Z ---------------------------------------------------------------------- 2022-09-27T16:07:54.2484055Z Ran 1 test in 2.159s 2022-09-27T16:07:54.2484171Z 2022-09-27T16:07:54.2484244Z OK (skipped=1) 2022-09-27T16:07:54.2484337Z 2022-09-27T16:07:54.2484430Z Generating XML reports... 2022-09-27T16:07:54.2528867Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160752.xml 2022-09-27T16:07:55.6462750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:55.6463120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:55.6470944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:55.6471293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:55.8026694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5n30n52 2022-09-27T16:07:55.8027401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5n30n52/_remote_module_non_scriptable.py 2022-09-27T16:07:56.0589055Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:07:56.0600971Z 2022-09-27T16:07:56.0601277Z Running tests... 2022-09-27T16:07:56.0601944Z ---------------------------------------------------------------------- 2022-09-27T16:07:56.3006305Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4805 2022-09-27T16:07:56.3033932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4806 2022-09-27T16:07:56.3062333Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4807 2022-09-27T16:07:56.3091794Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4808 2022-09-27T16:07:57.3845884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:57.3846481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:57.4208545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:57.4209177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:57.4623390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:57.4624015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:57.4994837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:57.4995450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:57.5069188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:57.5069840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:57.5424819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:57.5425416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:57.5483603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3cifkpwb 2022-09-27T16:07:57.5484941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3cifkpwb/_remote_module_non_scriptable.py 2022-09-27T16:07:57.5532979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:57.5533587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:57.5890925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:57.5891625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:57.6259028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphj1b4v2s 2022-09-27T16:07:57.6259860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphj1b4v2s/_remote_module_non_scriptable.py 2022-09-27T16:07:57.6688771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmlr1pvxh 2022-09-27T16:07:57.6689869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmlr1pvxh/_remote_module_non_scriptable.py 2022-09-27T16:07:57.7156328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitte6640 2022-09-27T16:07:57.7157138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitte6640/_remote_module_non_scriptable.py 2022-09-27T16:07:57.8090810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:07:57.8983934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:07:57.9255736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:07:57.9761642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:07:58.3147867Z skip: Need at least 2 CUDA devices (2.254s) 2022-09-27T16:07:58.3148171Z 2022-09-27T16:07:58.3148594Z ---------------------------------------------------------------------- 2022-09-27T16:07:58.3148850Z Ran 1 test in 2.255s 2022-09-27T16:07:58.3148951Z 2022-09-27T16:07:58.3149024Z OK (skipped=1) 2022-09-27T16:07:58.3149133Z 2022-09-27T16:07:58.3149218Z Generating XML reports... 2022-09-27T16:07:58.3195093Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160756.xml 2022-09-27T16:07:59.6769404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:07:59.6769754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:07:59.6779149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:07:59.6779507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:07:59.8412337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_6lx2qn 2022-09-27T16:07:59.8413654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_6lx2qn/_remote_module_non_scriptable.py 2022-09-27T16:08:00.0986857Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:00.0997033Z 2022-09-27T16:08:00.0997194Z Running tests... 2022-09-27T16:08:00.3389544Z ---------------------------------------------------------------------- 2022-09-27T16:08:00.3390392Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4860 2022-09-27T16:08:00.3416565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4861 2022-09-27T16:08:00.3443989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4862 2022-09-27T16:08:00.3472683Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4863 2022-09-27T16:08:01.4403928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:01.4404606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:01.4658755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:01.4659454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:01.4660324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:01.4660897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:01.4673765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:01.4674465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:01.4806379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:01.4806910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:01.5063322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:01.5063969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:01.5064685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:01.5065220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:01.5074979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:01.5075517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:01.6091773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxdcn4iz1 2022-09-27T16:08:01.6092562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxdcn4iz1/_remote_module_non_scriptable.py 2022-09-27T16:08:01.6360373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwc6azbj 2022-09-27T16:08:01.6361042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rnc_u5c 2022-09-27T16:08:01.6361653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpec1uimxz 2022-09-27T16:08:01.6362331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwc6azbj/_remote_module_non_scriptable.py 2022-09-27T16:08:01.6362955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rnc_u5c/_remote_module_non_scriptable.py 2022-09-27T16:08:01.6363669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpec1uimxz/_remote_module_non_scriptable.py 2022-09-27T16:08:01.8763061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:01.9037542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:01.9042537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:01.9056314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:02.2528490Z skip: Need at least 1 CUDA device (2.153s) 2022-09-27T16:08:02.2528768Z 2022-09-27T16:08:02.2529316Z ---------------------------------------------------------------------- 2022-09-27T16:08:02.2529618Z Ran 1 test in 2.153s 2022-09-27T16:08:02.2529734Z 2022-09-27T16:08:02.2529806Z OK (skipped=1) 2022-09-27T16:08:02.2529915Z 2022-09-27T16:08:02.2530000Z Generating XML reports... 2022-09-27T16:08:02.2573683Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160800.xml 2022-09-27T16:08:03.6589002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:03.6589421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:03.6597266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:03.6597867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:03.8169713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmparx8_2js 2022-09-27T16:08:03.8170595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmparx8_2js/_remote_module_non_scriptable.py 2022-09-27T16:08:04.0735835Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:04.0746036Z 2022-09-27T16:08:04.0746131Z Running tests... 2022-09-27T16:08:04.0746624Z ---------------------------------------------------------------------- 2022-09-27T16:08:04.3197891Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4915 2022-09-27T16:08:04.3225507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4916 2022-09-27T16:08:04.3252296Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4917 2022-09-27T16:08:04.3280815Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4918 2022-09-27T16:08:05.4314002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:05.4314834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:05.4355620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:05.4356234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:05.4356954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:05.4357533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:05.4459815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:05.4709314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:05.4710113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:05.4710629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:05.4746738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:05.4747377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:05.4748158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:05.4748775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:05.4836246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:05.4836857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:05.5974351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmymmoqnx 2022-09-27T16:08:05.5975175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmymmoqnx/_remote_module_non_scriptable.py 2022-09-27T16:08:05.6002079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwq2y4igw 2022-09-27T16:08:05.6002764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmcqjjnyl 2022-09-27T16:08:05.6003409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwq2y4igw/_remote_module_non_scriptable.py 2022-09-27T16:08:05.6005048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmcqjjnyl/_remote_module_non_scriptable.py 2022-09-27T16:08:05.6090328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9qn7hw4 2022-09-27T16:08:05.6091980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9qn7hw4/_remote_module_non_scriptable.py 2022-09-27T16:08:05.8552180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:05.8553516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:05.8563854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:05.8643395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:06.1332581Z skip: Need at least 2 CUDA devices (2.058s) 2022-09-27T16:08:06.1332767Z 2022-09-27T16:08:06.1333068Z ---------------------------------------------------------------------- 2022-09-27T16:08:06.1333307Z Ran 1 test in 2.059s 2022-09-27T16:08:06.1333422Z 2022-09-27T16:08:06.1333501Z OK (skipped=1) 2022-09-27T16:08:06.1333613Z 2022-09-27T16:08:06.1333698Z Generating XML reports... 2022-09-27T16:08:06.1380531Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160804.xml 2022-09-27T16:08:07.4917724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:07.4918123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:07.4926739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:07.4927137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:07.6527525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgfno5vq 2022-09-27T16:08:07.6528687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgfno5vq/_remote_module_non_scriptable.py 2022-09-27T16:08:07.9095591Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:07.9106074Z 2022-09-27T16:08:07.9106212Z Running tests... 2022-09-27T16:08:07.9106663Z ---------------------------------------------------------------------- 2022-09-27T16:08:08.1489095Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4970 2022-09-27T16:08:08.1515936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4971 2022-09-27T16:08:08.1543225Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4972 2022-09-27T16:08:08.1572673Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4973 2022-09-27T16:08:09.2843137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:09.2843856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:09.2844650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:09.2845293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:09.2846058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:09.2846684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:09.3107980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:09.3108627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:09.3230659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:09.3231356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:09.3232076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:09.3232895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:09.3237831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:09.3238662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:09.3473888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:09.3474752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:09.4502577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7cujnkbp 2022-09-27T16:08:09.4503351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7cujnkbp/_remote_module_non_scriptable.py 2022-09-27T16:08:09.4520764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84ahfyh4 2022-09-27T16:08:09.4522051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84ahfyh4/_remote_module_non_scriptable.py 2022-09-27T16:08:09.4680848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbv3748eu 2022-09-27T16:08:09.4681658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbv3748eu/_remote_module_non_scriptable.py 2022-09-27T16:08:09.4776982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_a6z6m0q 2022-09-27T16:08:09.4778401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_a6z6m0q/_remote_module_non_scriptable.py 2022-09-27T16:08:09.7055568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:09.7095652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:09.7249769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:09.7352791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:10.0626967Z skip: Need at least 2 CUDA devices (2.152s) 2022-09-27T16:08:10.0627254Z 2022-09-27T16:08:10.0627732Z ---------------------------------------------------------------------- 2022-09-27T16:08:10.0628118Z Ran 1 test in 2.152s 2022-09-27T16:08:10.0628303Z 2022-09-27T16:08:10.0628421Z OK (skipped=1) 2022-09-27T16:08:10.0628588Z 2022-09-27T16:08:10.0628703Z Generating XML reports... 2022-09-27T16:08:10.0673883Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160807.xml 2022-09-27T16:08:11.4638485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:11.4638843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:11.4647181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:11.4647556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:11.6219255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6oi8yboq 2022-09-27T16:08:11.6220040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6oi8yboq/_remote_module_non_scriptable.py 2022-09-27T16:08:11.8781577Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:11.8791709Z 2022-09-27T16:08:11.8791923Z Running tests... 2022-09-27T16:08:11.8792361Z ---------------------------------------------------------------------- 2022-09-27T16:08:12.1232633Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5025 2022-09-27T16:08:12.1260801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5026 2022-09-27T16:08:12.1287868Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5027 2022-09-27T16:08:12.1317124Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5028 2022-09-27T16:08:13.2776786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:13.2777740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:13.2822363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:13.2822987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:13.2962668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:13.2963302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:13.3133937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:13.3134567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:13.3150588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:13.3151307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:13.3213044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:13.3213679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:13.3360762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:13.3361416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:13.3517848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:13.3518455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:13.4433622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2vv0t_ka 2022-09-27T16:08:13.4434364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2vv0t_ka/_remote_module_non_scriptable.py 2022-09-27T16:08:13.4483537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6zgox4r 2022-09-27T16:08:13.4484978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6zgox4r/_remote_module_non_scriptable.py 2022-09-27T16:08:13.4642686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1j64fjy1 2022-09-27T16:08:13.4643826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1j64fjy1/_remote_module_non_scriptable.py 2022-09-27T16:08:13.4788453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9onbcee 2022-09-27T16:08:13.4790407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9onbcee/_remote_module_non_scriptable.py 2022-09-27T16:08:13.7057322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:13.7067568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:13.7392364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:13.7473850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:14.1373276Z skip: Need at least 1 CUDA device (2.258s) 2022-09-27T16:08:14.1373550Z 2022-09-27T16:08:14.1373967Z ---------------------------------------------------------------------- 2022-09-27T16:08:14.1374213Z Ran 1 test in 2.258s 2022-09-27T16:08:14.1374326Z 2022-09-27T16:08:14.1374399Z OK (skipped=1) 2022-09-27T16:08:14.1374506Z 2022-09-27T16:08:14.1374581Z Generating XML reports... 2022-09-27T16:08:14.1418528Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160811.xml 2022-09-27T16:08:15.5146444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:15.5147081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:15.5155277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:15.5155640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:15.6744037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgyvz9cu0 2022-09-27T16:08:15.6744712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgyvz9cu0/_remote_module_non_scriptable.py 2022-09-27T16:08:15.9292589Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:15.9302743Z 2022-09-27T16:08:15.9302882Z Running tests... 2022-09-27T16:08:15.9303491Z ---------------------------------------------------------------------- 2022-09-27T16:08:16.1758941Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5080 2022-09-27T16:08:16.1786255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5081 2022-09-27T16:08:16.1813735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5082 2022-09-27T16:08:16.1842378Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5083 2022-09-27T16:08:17.3355668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:17.3356272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:17.3458953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:17.3459630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:17.3721476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:17.3722165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:17.3823039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:17.3823707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:17.3824496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:17.3825045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:17.4025639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:17.4026276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:17.4195730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:17.4196383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:17.4407515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:17.4408188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:17.4991399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyrf25_4 2022-09-27T16:08:17.4992209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyrf25_4/_remote_module_non_scriptable.py 2022-09-27T16:08:17.5092938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyav1u4fd 2022-09-27T16:08:17.5094283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyav1u4fd/_remote_module_non_scriptable.py 2022-09-27T16:08:17.5459615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfuoht5rm 2022-09-27T16:08:17.5460442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfuoht5rm/_remote_module_non_scriptable.py 2022-09-27T16:08:17.5680250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7om8d7y 2022-09-27T16:08:17.5681082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7om8d7y/_remote_module_non_scriptable.py 2022-09-27T16:08:17.7599462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:17.7879476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:17.8066273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:17.8327385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:18.1899378Z skip: Need at least 2 CUDA devices (2.259s) 2022-09-27T16:08:18.1899717Z 2022-09-27T16:08:18.1900151Z ---------------------------------------------------------------------- 2022-09-27T16:08:18.1900415Z Ran 1 test in 2.260s 2022-09-27T16:08:18.1900518Z 2022-09-27T16:08:18.1900592Z OK (skipped=1) 2022-09-27T16:08:18.1900700Z 2022-09-27T16:08:18.1900785Z Generating XML reports... 2022-09-27T16:08:18.1946126Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160815.xml 2022-09-27T16:08:19.5730712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:19.5731095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:19.5739608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:19.5739954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:19.7358011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdayef4t_ 2022-09-27T16:08:19.7358872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdayef4t_/_remote_module_non_scriptable.py 2022-09-27T16:08:19.9914175Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:19.9925298Z 2022-09-27T16:08:19.9925428Z Running tests... 2022-09-27T16:08:19.9926145Z ---------------------------------------------------------------------- 2022-09-27T16:08:20.2321463Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5135 2022-09-27T16:08:20.2348050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5136 2022-09-27T16:08:20.2375971Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5137 2022-09-27T16:08:20.2405175Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5138 2022-09-27T16:08:21.3698098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:21.3698752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:21.4037622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:21.4038295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:21.4307599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:21.4308266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:21.4670184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:21.4670874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:21.5251767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:21.5252549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:21.5302560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4_8a209 2022-09-27T16:08:21.5304057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4_8a209/_remote_module_non_scriptable.py 2022-09-27T16:08:21.5488031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:21.5488731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:21.5621719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:21.5622409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:21.5849827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:21.5850554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:21.5958139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpowu6agn1 2022-09-27T16:08:21.5958897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpowu6agn1/_remote_module_non_scriptable.py 2022-09-27T16:08:21.6878981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcv8geclu 2022-09-27T16:08:21.6879845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcv8geclu/_remote_module_non_scriptable.py 2022-09-27T16:08:21.7094841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ctjchi_ 2022-09-27T16:08:21.7095566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ctjchi_/_remote_module_non_scriptable.py 2022-09-27T16:08:21.7901450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:21.8553834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:21.9680042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:21.9912936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:22.2458791Z skip: Need at least 2 CUDA devices (2.253s) 2022-09-27T16:08:22.2459113Z 2022-09-27T16:08:22.2459541Z ---------------------------------------------------------------------- 2022-09-27T16:08:22.2459792Z Ran 1 test in 2.253s 2022-09-27T16:08:22.2459911Z 2022-09-27T16:08:22.2459971Z OK (skipped=1) 2022-09-27T16:08:22.2460083Z 2022-09-27T16:08:22.2460169Z Generating XML reports... 2022-09-27T16:08:22.2504050Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160819.xml 2022-09-27T16:08:23.6120941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:23.6121558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:23.6130829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:23.6131235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:23.7700154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbt5m01y 2022-09-27T16:08:23.7700877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbt5m01y/_remote_module_non_scriptable.py 2022-09-27T16:08:24.0269001Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:24.0279647Z 2022-09-27T16:08:24.0279881Z Running tests... 2022-09-27T16:08:24.0280472Z ---------------------------------------------------------------------- 2022-09-27T16:08:24.2722076Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5190 2022-09-27T16:08:24.2750300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5191 2022-09-27T16:08:24.2777611Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5192 2022-09-27T16:08:24.2806639Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5193 2022-09-27T16:08:25.4170800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:25.4171472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:25.4172293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:25.4172916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:25.4407782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:25.4408460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:25.4511207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:25.4511738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:25.4557644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:25.4558279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:25.4561268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:25.4561894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:25.4802188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:25.4802868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:25.4902778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:25.4903442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:25.5835266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdj7pj8wl 2022-09-27T16:08:25.5836027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdj7pj8wl/_remote_module_non_scriptable.py 2022-09-27T16:08:25.5837152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplahl0r_3 2022-09-27T16:08:25.5840146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplahl0r_3/_remote_module_non_scriptable.py 2022-09-27T16:08:25.6071827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatrw5ij_ 2022-09-27T16:08:25.6072561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatrw5ij_/_remote_module_non_scriptable.py 2022-09-27T16:08:25.6177303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyyw9fqjz 2022-09-27T16:08:25.6178382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyyw9fqjz/_remote_module_non_scriptable.py 2022-09-27T16:08:25.8415572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:25.8431954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:25.8648603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:25.8747486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:26.1859958Z skip: Need at least 2 CUDA devices (2.158s) 2022-09-27T16:08:26.1860242Z 2022-09-27T16:08:26.1861070Z ---------------------------------------------------------------------- 2022-09-27T16:08:26.1861321Z Ran 1 test in 2.158s 2022-09-27T16:08:26.1861540Z 2022-09-27T16:08:26.1861611Z OK (skipped=1) 2022-09-27T16:08:26.1861717Z 2022-09-27T16:08:26.1861801Z Generating XML reports... 2022-09-27T16:08:26.1905444Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160824.xml 2022-09-27T16:08:27.5597099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:27.5597458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:27.5606192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:27.5606548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:27.7200083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9yquji9_ 2022-09-27T16:08:27.7201592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9yquji9_/_remote_module_non_scriptable.py 2022-09-27T16:08:27.9775561Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:27.9785699Z 2022-09-27T16:08:27.9785936Z Running tests... 2022-09-27T16:08:27.9786377Z ---------------------------------------------------------------------- 2022-09-27T16:08:28.2208187Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5245 2022-09-27T16:08:28.2234792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5246 2022-09-27T16:08:28.2261949Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5247 2022-09-27T16:08:28.2290923Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5248 2022-09-27T16:08:29.3906243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:29.3906920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:29.4137020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:29.4137516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:29.4149633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:29.4150272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:29.4285290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:29.4285822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:29.4411510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:29.4412011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:29.4525124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:29.4525850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:29.4537652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:29.4538235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:29.4761004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:29.4761588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:29.5548435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph91fhqbr 2022-09-27T16:08:29.5549698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph91fhqbr/_remote_module_non_scriptable.py 2022-09-27T16:08:29.5793558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9wfg_bl 2022-09-27T16:08:29.5795121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9wfg_bl/_remote_module_non_scriptable.py 2022-09-27T16:08:29.5795848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6nd7c9u_ 2022-09-27T16:08:29.5797456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6nd7c9u_/_remote_module_non_scriptable.py 2022-09-27T16:08:29.6012187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2t3_tj0r 2022-09-27T16:08:29.6013271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2t3_tj0r/_remote_module_non_scriptable.py 2022-09-27T16:08:29.8131245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:29.8368370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:29.8385534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:29.8604626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:30.1344768Z skip: Need at least 1 CUDA device (2.155s) 2022-09-27T16:08:30.1344956Z 2022-09-27T16:08:30.1345253Z ---------------------------------------------------------------------- 2022-09-27T16:08:30.1345508Z Ran 1 test in 2.156s 2022-09-27T16:08:30.1345622Z 2022-09-27T16:08:30.1345682Z OK (skipped=1) 2022-09-27T16:08:30.1345788Z 2022-09-27T16:08:30.1345874Z Generating XML reports... 2022-09-27T16:08:30.1390799Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160827.xml 2022-09-27T16:08:31.5027735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:31.5028112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:31.5036317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:31.5036666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:31.6612173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4q9q3hkp 2022-09-27T16:08:31.6613244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4q9q3hkp/_remote_module_non_scriptable.py 2022-09-27T16:08:31.9163507Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:31.9173837Z 2022-09-27T16:08:31.9174266Z Running tests... 2022-09-27T16:08:31.9174673Z ---------------------------------------------------------------------- 2022-09-27T16:08:32.1612781Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5300 2022-09-27T16:08:32.1640565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5301 2022-09-27T16:08:32.1667441Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5302 2022-09-27T16:08:32.1696213Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5303 2022-09-27T16:08:33.3362897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:33.3363505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:33.3474464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:33.3475296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:33.3675204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:33.3675756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:33.3723111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:33.3723689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:33.3848271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:33.3848818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:33.4047596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:33.4048212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:33.4079658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:33.4080324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:33.4449536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:33.4450225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:33.4993351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv96x15_s 2022-09-27T16:08:33.4994100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv96x15_s/_remote_module_non_scriptable.py 2022-09-27T16:08:33.5122347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3t1fay91 2022-09-27T16:08:33.5123446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3t1fay91/_remote_module_non_scriptable.py 2022-09-27T16:08:33.5329123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgb16w6l5 2022-09-27T16:08:33.5329895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgb16w6l5/_remote_module_non_scriptable.py 2022-09-27T16:08:33.5709373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa7onssh_ 2022-09-27T16:08:33.5710405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa7onssh_/_remote_module_non_scriptable.py 2022-09-27T16:08:33.7593008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:33.7691898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:33.7947684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:33.8313128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:34.1752093Z skip: Need at least 2 CUDA devices (2.257s) 2022-09-27T16:08:34.1752399Z 2022-09-27T16:08:34.1752739Z ---------------------------------------------------------------------- 2022-09-27T16:08:34.1753028Z Ran 1 test in 2.258s 2022-09-27T16:08:34.1753143Z 2022-09-27T16:08:34.1753216Z OK (skipped=1) 2022-09-27T16:08:34.1753336Z 2022-09-27T16:08:34.1753421Z Generating XML reports... 2022-09-27T16:08:34.1796661Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160831.xml 2022-09-27T16:08:35.5399674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:35.5400059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:35.5407922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:35.5408267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:35.6956550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3h66q7j2 2022-09-27T16:08:35.6957837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3h66q7j2/_remote_module_non_scriptable.py 2022-09-27T16:08:35.9524284Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:35.9534505Z 2022-09-27T16:08:35.9534975Z Running tests... 2022-09-27T16:08:35.9535377Z ---------------------------------------------------------------------- 2022-09-27T16:08:36.1845469Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81962 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.231s) 2022-09-27T16:08:36.1846006Z 2022-09-27T16:08:36.1846216Z ---------------------------------------------------------------------- 2022-09-27T16:08:36.1846479Z Ran 1 test in 0.231s 2022-09-27T16:08:36.1846607Z 2022-09-27T16:08:36.1846683Z OK (skipped=1) 2022-09-27T16:08:36.1846793Z 2022-09-27T16:08:36.1846864Z Generating XML reports... 2022-09-27T16:08:36.1874489Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160835.xml 2022-09-27T16:08:37.5111478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:37.5111861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:37.5120396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:37.5120757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:37.6712883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzd0vkmt 2022-09-27T16:08:37.6713354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzd0vkmt/_remote_module_non_scriptable.py 2022-09-27T16:08:37.9290703Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:37.9301964Z 2022-09-27T16:08:37.9302144Z Running tests... 2022-09-27T16:08:37.9302489Z ---------------------------------------------------------------------- 2022-09-27T16:08:38.1745738Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5365 2022-09-27T16:08:38.1772315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5366 2022-09-27T16:08:38.1799837Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5367 2022-09-27T16:08:38.1828412Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5368 2022-09-27T16:08:39.3187084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:39.3187771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:39.3307370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:39.3308032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:39.3358354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:39.3358855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:39.3536585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:39.3537299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:39.3570529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:39.3571369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:39.3705672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:39.3706710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:39.3754714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:39.3755362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:39.3983626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:39.3984305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:39.4862129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrjwuidi 2022-09-27T16:08:39.4862992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrjwuidi/_remote_module_non_scriptable.py 2022-09-27T16:08:39.4977958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmf43mipa 2022-09-27T16:08:39.4978802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmf43mipa/_remote_module_non_scriptable.py 2022-09-27T16:08:39.5017579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpijhmqg3y 2022-09-27T16:08:39.5019085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpijhmqg3y/_remote_module_non_scriptable.py 2022-09-27T16:08:39.5248824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1kmogytz 2022-09-27T16:08:39.5249639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1kmogytz/_remote_module_non_scriptable.py 2022-09-27T16:08:39.7440486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:39.7563500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:39.7569579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:39.7912912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:40.0883754Z skip: Need at least 2 CUDA devices (2.158s) 2022-09-27T16:08:40.0884054Z 2022-09-27T16:08:40.0884489Z ---------------------------------------------------------------------- 2022-09-27T16:08:40.0884737Z Ran 1 test in 2.158s 2022-09-27T16:08:40.0884860Z 2022-09-27T16:08:40.0884920Z OK (skipped=1) 2022-09-27T16:08:40.0885030Z 2022-09-27T16:08:40.0885115Z Generating XML reports... 2022-09-27T16:08:40.0930583Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160837.xml 2022-09-27T16:08:41.4861559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:41.4862228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:41.4871304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:41.4871942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:41.6452652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppbclmple 2022-09-27T16:08:41.6453496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppbclmple/_remote_module_non_scriptable.py 2022-09-27T16:08:41.9013357Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:41.9023984Z 2022-09-27T16:08:41.9024355Z Running tests... 2022-09-27T16:08:41.9024971Z ---------------------------------------------------------------------- 2022-09-27T16:08:42.1472070Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5420 2022-09-27T16:08:42.1499808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5421 2022-09-27T16:08:42.1527271Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5422 2022-09-27T16:08:42.1556347Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5423 2022-09-27T16:08:43.2402112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:43.2402769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:43.2403575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:43.2404210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:43.2774953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:43.2775558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:43.2776356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:43.2776904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:43.2799030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:43.2799601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:43.2839172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:43.2839792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:43.3172222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:43.3173026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:43.3213959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:43.3214574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:43.4055400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd89f95x0 2022-09-27T16:08:43.4056201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qi1zqz0 2022-09-27T16:08:43.4056948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd89f95x0/_remote_module_non_scriptable.py 2022-09-27T16:08:43.4057642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qi1zqz0/_remote_module_non_scriptable.py 2022-09-27T16:08:43.4445870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpar1v457i 2022-09-27T16:08:43.4447079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpar1v457i/_remote_module_non_scriptable.py 2022-09-27T16:08:43.4471547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphe899j6z 2022-09-27T16:08:43.4473018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphe899j6z/_remote_module_non_scriptable.py 2022-09-27T16:08:43.6679538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:43.6691022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:43.7051332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:43.7252703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:44.0611749Z skip: Need at least 1 CUDA device (2.158s) 2022-09-27T16:08:44.0612026Z 2022-09-27T16:08:44.0612305Z ---------------------------------------------------------------------- 2022-09-27T16:08:44.0612634Z Ran 1 test in 2.159s 2022-09-27T16:08:44.0612750Z 2022-09-27T16:08:44.0612821Z OK (skipped=1) 2022-09-27T16:08:44.0613156Z 2022-09-27T16:08:44.0613297Z Generating XML reports... 2022-09-27T16:08:44.0657161Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160841.xml 2022-09-27T16:08:45.4466121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:45.4466497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:45.4474221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:45.4482277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:45.6058048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeki3ejw9 2022-09-27T16:08:45.6058497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeki3ejw9/_remote_module_non_scriptable.py 2022-09-27T16:08:45.8614379Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:45.8625201Z 2022-09-27T16:08:45.8625308Z Running tests... 2022-09-27T16:08:45.8625836Z ---------------------------------------------------------------------- 2022-09-27T16:08:46.1053144Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5475 2022-09-27T16:08:46.1080851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5476 2022-09-27T16:08:46.1107897Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5477 2022-09-27T16:08:46.1138108Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5478 2022-09-27T16:08:47.2096178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:47.2096800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:47.2339384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:47.2340055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:47.2418570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:47.2419240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:47.2420050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:47.2420666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:47.2462943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:47.2463638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:47.2716924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:47.2717639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:47.2798135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:47.2798752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:47.2799461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:47.2800000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:47.3728850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnfqpg_g 2022-09-27T16:08:47.3730151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnfqpg_g/_remote_module_non_scriptable.py 2022-09-27T16:08:47.3985296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ljvf0zq 2022-09-27T16:08:47.3986009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ljvf0zq/_remote_module_non_scriptable.py 2022-09-27T16:08:47.4075680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3dvuspk 2022-09-27T16:08:47.4076559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bwtk9v2 2022-09-27T16:08:47.4077301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3dvuspk/_remote_module_non_scriptable.py 2022-09-27T16:08:47.4078071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bwtk9v2/_remote_module_non_scriptable.py 2022-09-27T16:08:47.6334008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:47.6685820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:47.6697375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:47.6748178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:48.0191819Z skip: Need at least 1 CUDA device (2.156s) 2022-09-27T16:08:48.0192617Z 2022-09-27T16:08:48.0194691Z ---------------------------------------------------------------------- 2022-09-27T16:08:48.0195134Z Ran 1 test in 2.157s 2022-09-27T16:08:48.0195315Z 2022-09-27T16:08:48.0195426Z OK (skipped=1) 2022-09-27T16:08:48.0195611Z 2022-09-27T16:08:48.0195746Z Generating XML reports... 2022-09-27T16:08:48.0240749Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160845.xml 2022-09-27T16:08:49.3901110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:49.3901498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:49.3909341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:49.3909870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:49.5451074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7vwcpda 2022-09-27T16:08:49.5451905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7vwcpda/_remote_module_non_scriptable.py 2022-09-27T16:08:49.8030487Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:49.8041404Z 2022-09-27T16:08:49.8041505Z Running tests... 2022-09-27T16:08:49.8042074Z ---------------------------------------------------------------------- 2022-09-27T16:08:50.0447238Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5530 2022-09-27T16:08:50.0474034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5531 2022-09-27T16:08:50.0500861Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5532 2022-09-27T16:08:50.0530155Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5533 2022-09-27T16:08:51.1027634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:51.1028320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:51.1386702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:51.1387284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:51.1590763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:51.1591401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:51.1756273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:51.1757009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:51.1797794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:51.1798397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:51.1969910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:51.1970525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:51.2150052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:51.2150749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:51.2192952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:51.2193582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:51.2664098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcu_89j__ 2022-09-27T16:08:51.2664770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcu_89j__/_remote_module_non_scriptable.py 2022-09-27T16:08:51.3240020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnobhflwz 2022-09-27T16:08:51.3240986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnobhflwz/_remote_module_non_scriptable.py 2022-09-27T16:08:51.3412342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpgpzxi7 2022-09-27T16:08:51.3413792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpgpzxi7/_remote_module_non_scriptable.py 2022-09-27T16:08:51.3462490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_gllqo2v 2022-09-27T16:08:51.3464069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_gllqo2v/_remote_module_non_scriptable.py 2022-09-27T16:08:51.5233539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:51.5806596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:51.5997273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:51.6033297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:51.9584939Z skip: Need at least 2 CUDA devices (2.154s) 2022-09-27T16:08:51.9585262Z 2022-09-27T16:08:51.9585774Z ---------------------------------------------------------------------- 2022-09-27T16:08:51.9586031Z Ran 1 test in 2.154s 2022-09-27T16:08:51.9586150Z 2022-09-27T16:08:51.9586211Z OK (skipped=1) 2022-09-27T16:08:51.9586320Z 2022-09-27T16:08:51.9586421Z Generating XML reports... 2022-09-27T16:08:51.9634388Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160849.xml 2022-09-27T16:08:53.3372887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:53.3373258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:53.3380199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:53.3380577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:53.4924946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphktp8hsv 2022-09-27T16:08:53.4926092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphktp8hsv/_remote_module_non_scriptable.py 2022-09-27T16:08:53.7488353Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:53.7499427Z 2022-09-27T16:08:53.7499830Z Running tests... 2022-09-27T16:08:53.7500381Z ---------------------------------------------------------------------- 2022-09-27T16:08:53.9924165Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5585 2022-09-27T16:08:53.9951056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5586 2022-09-27T16:08:53.9977989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5587 2022-09-27T16:08:54.0006573Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5588 2022-09-27T16:08:55.1059396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:55.1060030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:55.1139741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:55.1140370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:55.1151244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:55.1151896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:55.1329520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:55.1330044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:55.1454218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:55.1454907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:55.1542180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:55.1542782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:55.1549556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:55.1550220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:55.1709875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:55.1710505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:55.2719084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzn0v0oi5 2022-09-27T16:08:55.2719892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzn0v0oi5/_remote_module_non_scriptable.py 2022-09-27T16:08:55.2808877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcerb23_d 2022-09-27T16:08:55.2810463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcerb23_d/_remote_module_non_scriptable.py 2022-09-27T16:08:55.2811111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsaxko1xf 2022-09-27T16:08:55.2813585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsaxko1xf/_remote_module_non_scriptable.py 2022-09-27T16:08:55.2981330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0f5eljsz 2022-09-27T16:08:55.2982271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0f5eljsz/_remote_module_non_scriptable.py 2022-09-27T16:08:55.5326218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:55.5391767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:55.5396256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:55.5608535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:55.9061530Z skip: Need at least 2 CUDA devices (2.156s) 2022-09-27T16:08:55.9062132Z 2022-09-27T16:08:55.9062575Z ---------------------------------------------------------------------- 2022-09-27T16:08:55.9062971Z Ran 1 test in 2.156s 2022-09-27T16:08:55.9063142Z 2022-09-27T16:08:55.9063255Z OK (skipped=1) 2022-09-27T16:08:55.9063422Z 2022-09-27T16:08:55.9063563Z Generating XML reports... 2022-09-27T16:08:55.9109779Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160853.xml 2022-09-27T16:08:57.2969219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:57.2969640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:57.2978608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:57.2978968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:57.4568878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuk2l4s4t 2022-09-27T16:08:57.4570092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuk2l4s4t/_remote_module_non_scriptable.py 2022-09-27T16:08:57.7197101Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:08:57.7207276Z 2022-09-27T16:08:57.7207394Z Running tests... 2022-09-27T16:08:57.7207963Z ---------------------------------------------------------------------- 2022-09-27T16:08:57.9605565Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5640 2022-09-27T16:08:57.9632350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5641 2022-09-27T16:08:57.9659991Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5642 2022-09-27T16:08:57.9688980Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5643 2022-09-27T16:08:59.1089611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:59.1090212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:59.1187869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:59.1188534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:59.1416098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:59.1416592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:59.1470177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:59.1470927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:59.1489210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:08:59.1489870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:08:59.1572409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:59.1573045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:59.1795589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:59.1796224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:59.1861911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:08:59.1862589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:08:59.2739690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdf3g0b_ 2022-09-27T16:08:59.2740693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdf3g0b_/_remote_module_non_scriptable.py 2022-09-27T16:08:59.2843447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55_n4gie 2022-09-27T16:08:59.2844560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55_n4gie/_remote_module_non_scriptable.py 2022-09-27T16:08:59.3072114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2llkak3b 2022-09-27T16:08:59.3072903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2llkak3b/_remote_module_non_scriptable.py 2022-09-27T16:08:59.3130220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppo5pb_qy 2022-09-27T16:08:59.3132251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppo5pb_qy/_remote_module_non_scriptable.py 2022-09-27T16:08:59.5353907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:08:59.5503921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:08:59.5695072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:08:59.5845059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:08:59.8742596Z skip: Need at least 2 CUDA devices (2.153s) 2022-09-27T16:08:59.8742857Z 2022-09-27T16:08:59.8743385Z ---------------------------------------------------------------------- 2022-09-27T16:08:59.8743756Z Ran 1 test in 2.153s 2022-09-27T16:08:59.8743873Z 2022-09-27T16:08:59.8743944Z OK (skipped=1) 2022-09-27T16:08:59.8744055Z 2022-09-27T16:08:59.8744127Z Generating XML reports... 2022-09-27T16:08:59.8787870Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160857.xml 2022-09-27T16:09:01.2342572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:01.2343189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:01.2351423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:01.2352039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:01.3880339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppzwo14ze 2022-09-27T16:09:01.3881314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppzwo14ze/_remote_module_non_scriptable.py 2022-09-27T16:09:01.6489300Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:09:01.6499971Z 2022-09-27T16:09:01.6500068Z Running tests... 2022-09-27T16:09:01.6501004Z ---------------------------------------------------------------------- 2022-09-27T16:09:01.8990893Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5695 2022-09-27T16:09:01.9019478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5696 2022-09-27T16:09:01.9048269Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5697 2022-09-27T16:09:01.9078674Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5698 2022-09-27T16:09:03.0320099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:03.0320746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:03.0336130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:03.0336694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:03.0573119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:03.0573735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:03.0708078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:03.0708706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:03.0726164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:03.0726791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:03.0813520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:03.0814040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:03.0947458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:03.0948110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:03.1177270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:03.1177942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:03.1972586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzr7r6wov 2022-09-27T16:09:03.1973317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzr7r6wov/_remote_module_non_scriptable.py 2022-09-27T16:09:03.1996589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2l15b3_1 2022-09-27T16:09:03.1998030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2l15b3_1/_remote_module_non_scriptable.py 2022-09-27T16:09:03.2408827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrbze55m 2022-09-27T16:09:03.2409564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrbze55m/_remote_module_non_scriptable.py 2022-09-27T16:09:03.2477139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyw_0qeb3 2022-09-27T16:09:03.2479432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyw_0qeb3/_remote_module_non_scriptable.py 2022-09-27T16:09:03.4530853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:03.4559409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:09:03.5449407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:09:03.5634802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:03.9132257Z skip: Need at least 1 CUDA device (2.263s) 2022-09-27T16:09:03.9132607Z 2022-09-27T16:09:03.9133037Z ---------------------------------------------------------------------- 2022-09-27T16:09:03.9133272Z Ran 1 test in 2.263s 2022-09-27T16:09:03.9133384Z 2022-09-27T16:09:03.9133456Z OK (skipped=1) 2022-09-27T16:09:03.9133565Z 2022-09-27T16:09:03.9133650Z Generating XML reports... 2022-09-27T16:09:03.9176034Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160901.xml 2022-09-27T16:09:05.2660944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:05.2661557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:05.2669596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:05.2670493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:05.4253493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpck27ebpa 2022-09-27T16:09:05.4254355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpck27ebpa/_remote_module_non_scriptable.py 2022-09-27T16:09:05.6802590Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:09:05.6812731Z 2022-09-27T16:09:05.6813039Z Running tests... 2022-09-27T16:09:05.6813687Z ---------------------------------------------------------------------- 2022-09-27T16:09:05.9226210Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5750 2022-09-27T16:09:05.9251907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5751 2022-09-27T16:09:05.9278998Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5752 2022-09-27T16:09:05.9308006Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5753 2022-09-27T16:09:07.0183200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:07.0183808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:07.0242357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:07.0242976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:07.0424770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:07.0425452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:07.0555599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:07.0556146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:07.0619915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:07.0620463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:07.0686749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:07.0687381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:07.0802090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:07.0802699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:07.1065122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:07.1065811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:07.1825402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9ss4zha 2022-09-27T16:09:07.1826214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9ss4zha/_remote_module_non_scriptable.py 2022-09-27T16:09:07.1883009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rgv26_7 2022-09-27T16:09:07.1884488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rgv26_7/_remote_module_non_scriptable.py 2022-09-27T16:09:07.2074226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3hwz02i 2022-09-27T16:09:07.2076424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3hwz02i/_remote_module_non_scriptable.py 2022-09-27T16:09:07.2328224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4p0it8sv 2022-09-27T16:09:07.2329774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4p0it8sv/_remote_module_non_scriptable.py 2022-09-27T16:09:07.4452273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:07.4848966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:09:07.4868616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:07.4925351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:09:07.8364125Z skip: Need at least 2 CUDA devices (2.155s) 2022-09-27T16:09:07.8364432Z 2022-09-27T16:09:07.8364873Z ---------------------------------------------------------------------- 2022-09-27T16:09:07.8365139Z Ran 1 test in 2.155s 2022-09-27T16:09:07.8365241Z 2022-09-27T16:09:07.8365312Z OK (skipped=1) 2022-09-27T16:09:07.8365419Z 2022-09-27T16:09:07.8365503Z Generating XML reports... 2022-09-27T16:09:07.8410890Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160905.xml 2022-09-27T16:09:09.2012108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:09.2012506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:09.2021228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:09.2021577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:09.3617633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpus5x3b7g 2022-09-27T16:09:09.3619922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpus5x3b7g/_remote_module_non_scriptable.py 2022-09-27T16:09:09.6217063Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:09:09.6227047Z 2022-09-27T16:09:09.6227153Z Running tests... 2022-09-27T16:09:09.6227746Z ---------------------------------------------------------------------- 2022-09-27T16:09:09.8668344Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5805 2022-09-27T16:09:09.8696049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5806 2022-09-27T16:09:09.8723520Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5807 2022-09-27T16:09:09.8752693Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5808 2022-09-27T16:09:11.0215129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:11.0215785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:11.0266162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:11.0266764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:11.0282640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:11.0283226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:11.0348493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:11.0349143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:11.0596784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:11.0597297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:11.0653192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:11.0654183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:11.0675587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:11.0676475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:11.0732108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:11.0732768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:11.1866262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdk4i07n 2022-09-27T16:09:11.1868675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdk4i07n/_remote_module_non_scriptable.py 2022-09-27T16:09:11.1938165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpwtvdzs 2022-09-27T16:09:11.1940517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpwtvdzs/_remote_module_non_scriptable.py 2022-09-27T16:09:11.1945012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcqvp6qx9 2022-09-27T16:09:11.1947321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcqvp6qx9/_remote_module_non_scriptable.py 2022-09-27T16:09:11.1998636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuv8onhv7 2022-09-27T16:09:11.2000697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuv8onhv7/_remote_module_non_scriptable.py 2022-09-27T16:09:11.4544012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:09:11.4580948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:09:11.4622800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:11.4699211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:11.7807856Z skip: Need at least 2 CUDA devices (2.158s) 2022-09-27T16:09:11.7808184Z 2022-09-27T16:09:11.7808675Z ---------------------------------------------------------------------- 2022-09-27T16:09:11.7808934Z Ran 1 test in 2.158s 2022-09-27T16:09:11.7809053Z 2022-09-27T16:09:11.7809112Z OK (skipped=1) 2022-09-27T16:09:11.7809223Z 2022-09-27T16:09:11.7809491Z Generating XML reports... 2022-09-27T16:09:11.7853672Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160909.xml 2022-09-27T16:09:13.1554035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:13.1554398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:13.1562606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:13.1562986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:13.3106946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyx66yx70 2022-09-27T16:09:13.3108079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyx66yx70/_remote_module_non_scriptable.py 2022-09-27T16:09:13.5691235Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:09:13.5701838Z 2022-09-27T16:09:13.5701929Z Running tests... 2022-09-27T16:09:13.5702461Z ---------------------------------------------------------------------- 2022-09-27T16:09:13.8149839Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5860 2022-09-27T16:09:13.8176190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5861 2022-09-27T16:09:13.8204192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5862 2022-09-27T16:09:13.8232885Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5863 2022-09-27T16:09:15.0028920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:15.0029925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:15.0326718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:15.0327349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:15.0414526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:15.0415115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:15.0432043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:15.0432621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:15.0662729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:15.0663452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:15.0718080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:15.0718751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:15.0823205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:15.0823818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:15.1032312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:15.1032988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:15.1689724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpydolcoib 2022-09-27T16:09:15.1690525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpydolcoib/_remote_module_non_scriptable.py 2022-09-27T16:09:15.1990894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5p6vpbcj 2022-09-27T16:09:15.1991700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5p6vpbcj/_remote_module_non_scriptable.py 2022-09-27T16:09:15.2109811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9ynnjl5 2022-09-27T16:09:15.2111489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9ynnjl5/_remote_module_non_scriptable.py 2022-09-27T16:09:15.2431322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahy_gyv5 2022-09-27T16:09:15.2432958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahy_gyv5/_remote_module_non_scriptable.py 2022-09-27T16:09:15.4274177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:09:15.4581991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:15.4927320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:15.5267011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:09:15.8322710Z skip: Need at least 2 CUDA devices (2.262s) 2022-09-27T16:09:15.8323012Z 2022-09-27T16:09:15.8323419Z ---------------------------------------------------------------------- 2022-09-27T16:09:15.8323668Z Ran 1 test in 2.262s 2022-09-27T16:09:15.8323788Z 2022-09-27T16:09:15.8323850Z OK (skipped=1) 2022-09-27T16:09:15.8323962Z 2022-09-27T16:09:15.8324049Z Generating XML reports... 2022-09-27T16:09:15.8367922Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160913.xml 2022-09-27T16:09:17.2101736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:17.2102348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:17.2109888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:17.2110317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:17.3666041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj0euek9j 2022-09-27T16:09:17.3666928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj0euek9j/_remote_module_non_scriptable.py 2022-09-27T16:09:17.6220626Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:09:17.6231541Z 2022-09-27T16:09:17.6232006Z Running tests... 2022-09-27T16:09:17.6232422Z ---------------------------------------------------------------------- 2022-09-27T16:09:17.8674377Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5915 2022-09-27T16:09:17.8701029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5916 2022-09-27T16:09:17.8728018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5917 2022-09-27T16:09:17.8756772Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5918 2022-09-27T16:09:18.9410816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:18.9411532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:18.9598502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:18.9599102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:18.9772860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:18.9773614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:18.9774446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:18.9775030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:18.9870970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:18.9871628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:18.9972483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:18.9973023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:19.0148078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:19.0148732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:19.0245181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:19.0245860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:19.1030629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhij7rez 2022-09-27T16:09:19.1031514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhij7rez/_remote_module_non_scriptable.py 2022-09-27T16:09:19.1243841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_5oa1fj 2022-09-27T16:09:19.1245167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_5oa1fj/_remote_module_non_scriptable.py 2022-09-27T16:09:19.1403367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mnubqik 2022-09-27T16:09:19.1404672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mnubqik/_remote_module_non_scriptable.py 2022-09-27T16:09:19.1514097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9uhsce4g 2022-09-27T16:09:19.1515742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9uhsce4g/_remote_module_non_scriptable.py 2022-09-27T16:09:19.3593997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:09:19.3801360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:09:19.3973973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:19.4096291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:19.7811234Z skip: Need at least 1 CUDA device (2.158s) 2022-09-27T16:09:19.7811450Z 2022-09-27T16:09:19.7811774Z ---------------------------------------------------------------------- 2022-09-27T16:09:19.7812038Z Ran 1 test in 2.158s 2022-09-27T16:09:19.7812184Z 2022-09-27T16:09:19.7812285Z OK (skipped=1) 2022-09-27T16:09:19.7812399Z 2022-09-27T16:09:19.7812486Z Generating XML reports... 2022-09-27T16:09:19.7855580Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160917.xml 2022-09-27T16:09:21.1728656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:21.1729336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:21.1737756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:21.1738383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:21.3304445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzuop0p13 2022-09-27T16:09:21.3305433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzuop0p13/_remote_module_non_scriptable.py 2022-09-27T16:09:21.5932349Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:09:21.5943491Z 2022-09-27T16:09:21.5943946Z Running tests... 2022-09-27T16:09:21.5944370Z ---------------------------------------------------------------------- 2022-09-27T16:09:21.8433028Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5970 2022-09-27T16:09:21.8459304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5971 2022-09-27T16:09:21.8486718Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5972 2022-09-27T16:09:21.8514879Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5973 2022-09-27T16:09:22.9455452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:22.9456144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:22.9456947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:22.9457591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:22.9460433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:22.9461077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:22.9686761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:22.9687258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:22.9849063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:22.9849680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:22.9850570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:22.9851185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:22.9856045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:22.9856645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:23.0065712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:23.0066360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:23.1124373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4gon9dx7 2022-09-27T16:09:23.1125145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa7tszqht 2022-09-27T16:09:23.1126215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4gon9dx7/_remote_module_non_scriptable.py 2022-09-27T16:09:23.1126954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrc5plyo 2022-09-27T16:09:23.1127698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa7tszqht/_remote_module_non_scriptable.py 2022-09-27T16:09:23.1128661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrc5plyo/_remote_module_non_scriptable.py 2022-09-27T16:09:23.1322835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybmivyh9 2022-09-27T16:09:23.1324102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybmivyh9/_remote_module_non_scriptable.py 2022-09-27T16:09:23.3704169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:23.3814677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:23.3816083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:09:23.3898080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:09:23.7570333Z skip: Need at least 1 CUDA device (2.162s) 2022-09-27T16:09:23.7570646Z 2022-09-27T16:09:23.7571163Z ---------------------------------------------------------------------- 2022-09-27T16:09:23.7571459Z Ran 1 test in 2.163s 2022-09-27T16:09:23.7571576Z 2022-09-27T16:09:23.7571637Z OK (skipped=1) 2022-09-27T16:09:23.7571743Z 2022-09-27T16:09:23.7571830Z Generating XML reports... 2022-09-27T16:09:23.7618247Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160921.xml 2022-09-27T16:09:25.1485324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:25.1485724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:25.1492870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:25.1493268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:25.3084948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4z91tdi 2022-09-27T16:09:25.3086113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4z91tdi/_remote_module_non_scriptable.py 2022-09-27T16:09:25.5704444Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:09:25.5715041Z 2022-09-27T16:09:25.5715129Z Running tests... 2022-09-27T16:09:25.5715613Z ---------------------------------------------------------------------- 2022-09-27T16:09:25.8181714Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6025 2022-09-27T16:09:25.8209354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6026 2022-09-27T16:09:25.8236820Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6027 2022-09-27T16:09:25.8265669Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6028 2022-09-27T16:09:27.0225362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:27.0226022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:27.0267210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:27.0267833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:27.0316579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:27.0317217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:27.0431651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:27.0432328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:27.0614189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:27.0614979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:27.0657616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:27.0658132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:27.0707695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:27.0708348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:27.0804491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:27.0805112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:27.1877293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp86x5qk8m 2022-09-27T16:09:27.1877983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp86x5qk8m/_remote_module_non_scriptable.py 2022-09-27T16:09:27.1916807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8engv7c3 2022-09-27T16:09:27.1918417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8engv7c3/_remote_module_non_scriptable.py 2022-09-27T16:09:27.1971907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpu2xui_ 2022-09-27T16:09:27.1973162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpu2xui_/_remote_module_non_scriptable.py 2022-09-27T16:09:27.2065933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqg_6vo6 2022-09-27T16:09:27.2067288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqg_6vo6/_remote_module_non_scriptable.py 2022-09-27T16:09:27.4448892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:09:27.4494349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:27.4561408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:27.4640538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:09:27.8322442Z skip: Need at least 4 CUDA devices (2.260s) 2022-09-27T16:09:27.8322845Z 2022-09-27T16:09:27.8323899Z ---------------------------------------------------------------------- 2022-09-27T16:09:27.8324175Z Ran 1 test in 2.261s 2022-09-27T16:09:27.8324294Z 2022-09-27T16:09:27.8324442Z OK (skipped=1) 2022-09-27T16:09:27.8324555Z 2022-09-27T16:09:27.8324641Z Generating XML reports... 2022-09-27T16:09:27.8367446Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220927160925.xml 2022-09-27T16:09:29.2091911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:29.2092310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:29.2100685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:29.2101034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:29.3710836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcdeknv75 2022-09-27T16:09:29.3711943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcdeknv75/_remote_module_non_scriptable.py 2022-09-27T16:09:29.6301325Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:09:29.6311280Z 2022-09-27T16:09:29.6311415Z Running tests... 2022-09-27T16:09:29.6311832Z ---------------------------------------------------------------------- 2022-09-27T16:09:29.8755105Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6080 2022-09-27T16:09:29.8781817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6081 2022-09-27T16:09:29.8809265Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6082 2022-09-27T16:09:29.8838767Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6083 2022-09-27T16:09:31.0351625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:31.0352282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:31.0362469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:31.0363054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:31.0485911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:31.0486589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:31.0497920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:31.0498456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:31.0744793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:31.0745492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:31.0761606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:31.0762242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:31.0883329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:31.0884594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:31.0886195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:31.0886995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:31.2050661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpblwu68_v 2022-09-27T16:09:31.2051607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpblwu68_v/_remote_module_non_scriptable.py 2022-09-27T16:09:31.2052289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppq_j5sbx 2022-09-27T16:09:31.2055248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppq_j5sbx/_remote_module_non_scriptable.py 2022-09-27T16:09:31.2179257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgigfgvk 2022-09-27T16:09:31.2180085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgigfgvk/_remote_module_non_scriptable.py 2022-09-27T16:09:31.2300619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2pxvtzr8 2022-09-27T16:09:31.2301940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2pxvtzr8/_remote_module_non_scriptable.py 2022-09-27T16:09:31.4627177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:31.4643249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:09:31.4778299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:09:31.4861663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:31.7893024Z skip: Need at least 4 CUDA devices (2.158s) 2022-09-27T16:09:31.7893215Z 2022-09-27T16:09:31.7893518Z ---------------------------------------------------------------------- 2022-09-27T16:09:31.7893755Z Ran 1 test in 2.158s 2022-09-27T16:09:31.7893870Z 2022-09-27T16:09:31.7893942Z OK (skipped=1) 2022-09-27T16:09:31.7894051Z 2022-09-27T16:09:31.7894137Z Generating XML reports... 2022-09-27T16:09:31.7938498Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220927160929.xml 2022-09-27T16:09:33.1901737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:33.1902121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:33.1911062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:33.1911434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:33.3496236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgv4wsn20 2022-09-27T16:09:33.3497226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgv4wsn20/_remote_module_non_scriptable.py 2022-09-27T16:09:33.6067445Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-09-27T16:09:33.6078374Z 2022-09-27T16:09:33.6078538Z Running tests... 2022-09-27T16:09:33.6078879Z ---------------------------------------------------------------------- 2022-09-27T16:09:33.8499570Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6135 2022-09-27T16:09:33.8526098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6136 2022-09-27T16:09:33.8553161Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6137 2022-09-27T16:09:33.8583580Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6138 2022-09-27T16:09:34.9685332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:34.9878735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:34.9879544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:34.9906525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:34.9907525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:34.9908005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:34.9930658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:34.9931331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:35.0074968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:35.0075565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:35.0270812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:35.0271431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:35.0290243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:35.0290879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:35.0313377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:35.0314024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:35.1341841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8i7ybkf 2022-09-27T16:09:35.1342638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8i7ybkf/_remote_module_non_scriptable.py 2022-09-27T16:09:35.1545602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_y79k67 2022-09-27T16:09:35.1546646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_y79k67/_remote_module_non_scriptable.py 2022-09-27T16:09:35.1555426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphccqpr5o 2022-09-27T16:09:35.1557522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphccqpr5o/_remote_module_non_scriptable.py 2022-09-27T16:09:35.1583440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2kx4603 2022-09-27T16:09:35.1585341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2kx4603/_remote_module_non_scriptable.py 2022-09-27T16:09:35.3973514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:09:35.4185028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:35.4204754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:35.4498510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:09:35.7638283Z skip: Need at least 4 CUDA devices (2.156s) 2022-09-27T16:09:35.7638523Z 2022-09-27T16:09:35.7638838Z ---------------------------------------------------------------------- 2022-09-27T16:09:35.7639094Z Ran 1 test in 2.156s 2022-09-27T16:09:35.7639213Z 2022-09-27T16:09:35.7639286Z OK (skipped=1) 2022-09-27T16:09:35.7639396Z 2022-09-27T16:09:35.7639491Z Generating XML reports... 2022-09-27T16:09:35.7683192Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220927160933.xml 2022-09-27T16:09:36.3711179Z Running distributed/fsdp/test_fsdp_state_dict ... [2022-09-27 16:09:36.370712] 2022-09-27T16:09:36.3711821Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:09:36.370795] 2022-09-27T16:09:37.6174733Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict 2022-09-27T16:09:37.6192835Z 2022-09-27T16:09:37.6193231Z Running tests... 2022-09-27T16:09:37.6193662Z ---------------------------------------------------------------------- 2022-09-27T16:09:37.6213036Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:09:37.8682009Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6190 2022-09-27T16:09:37.8710022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6191 2022-09-27T16:09:38.9676531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:38.9676966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:38.9680141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:38.9680548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:38.9776938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:38.9777303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:38.9780902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:38.9781296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:39.0948321Z dist init r=1, world=2 2022-09-27T16:09:39.1044644Z dist init r=0, world=2 2022-09-27T16:09:39.1157437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:39.1158356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:39.1158772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:39.1159278Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:39.1163978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:39.1164537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:39.3743581Z skip: Need at least 2 CUDA devices (1.755s) 2022-09-27T16:09:39.3762328Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:09:39.3798470Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6218 2022-09-27T16:09:39.3827445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6219 2022-09-27T16:09:40.4814274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:40.4814815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:40.4815263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:40.4815588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:40.4817426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:40.4817851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:40.4819131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:40.4819462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:40.6080146Z dist init r=0, world=2 2022-09-27T16:09:40.6080487Z dist init r=1, world=2 2022-09-27T16:09:40.6188523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:40.6189683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:40.6190873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:40.6292262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:40.6296141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:40.6296546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:40.8860336Z skip: Need at least 2 CUDA devices (1.512s) 2022-09-27T16:09:40.8880071Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:09:40.8925282Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6246 2022-09-27T16:09:40.8959712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6247 2022-09-27T16:09:42.0076977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:42.0077421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:42.0079798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:42.0080161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:42.0154943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:42.0155288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:42.0158978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:42.0159353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:42.1349501Z dist init r=1, world=2 2022-09-27T16:09:42.1420996Z dist init r=0, world=2 2022-09-27T16:09:42.1629960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:42.1630643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:42.1631331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:42.1631865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:42.1635563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:42.1636194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:42.4993056Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:09:42.5014064Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:09:42.5059589Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6274 2022-09-27T16:09:42.5095019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6275 2022-09-27T16:09:43.6053087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:43.6053449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:43.6055993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:43.6056356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:43.6182196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:43.6182537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:43.6186188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:43.6186539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:43.7332123Z dist init r=1, world=2 2022-09-27T16:09:43.7456319Z dist init r=0, world=2 2022-09-27T16:09:43.7642320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:43.7642992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:43.7643600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:43.7644142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:43.7647791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:43.7648149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:44.0128644Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:09:44.0147488Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:09:44.0190775Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6302 2022-09-27T16:09:44.0223310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6303 2022-09-27T16:09:45.1148426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:45.1149076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:45.1152695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:45.1153090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:45.1181193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:45.1181712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:45.1185906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:45.1186482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:45.2406700Z dist init r=1, world=2 2022-09-27T16:09:45.2438931Z dist init r=0, world=2 2022-09-27T16:09:45.2648396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:45.2649102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:45.2650147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:45.2650922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:45.2653973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:45.2654627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:45.5256750Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:09:45.5276233Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:09:45.5314056Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6330 2022-09-27T16:09:45.5343294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6331 2022-09-27T16:09:46.6295015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:46.6295399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:46.6298509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:46.6298869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:46.6620822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:46.6621245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:46.6624949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:46.6625311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:46.7555090Z dist init r=0, world=2 2022-09-27T16:09:46.7886902Z dist init r=1, world=2 2022-09-27T16:09:46.7996375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:46.7996873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:46.7997494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:46.7998026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:46.8103329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:46.8104018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:47.1378059Z skip: Need at least 2 CUDA devices (1.612s) 2022-09-27T16:09:47.1397712Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:09:47.1444995Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6358 2022-09-27T16:09:47.1479372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6359 2022-09-27T16:09:48.2435305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:48.2435672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:48.2438569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:48.2438938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:48.2507062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:48.2507417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:48.2511641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:48.2512026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:48.3704923Z dist init r=0, world=2 2022-09-27T16:09:48.3780090Z dist init r=1, world=2 2022-09-27T16:09:48.3889210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:48.3889632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:48.3890324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:48.3890839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:48.3995170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:48.3995743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:48.6513475Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:09:48.6532491Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:09:48.6576842Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6386 2022-09-27T16:09:48.6609880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6387 2022-09-27T16:09:49.7679274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:49.7679909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:49.7681693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:49.7682293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:49.7700837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:49.7701365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:49.7705846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:49.7706235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:49.8945058Z dist init r=0, world=2 2022-09-27T16:09:49.8957210Z dist init r=1, world=2 2022-09-27T16:09:49.9065808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:49.9066234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:49.9067056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:49.9067632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:49.9071419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:49.9071796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:50.1643243Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:09:50.1662076Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:09:50.1708887Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6414 2022-09-27T16:09:50.1743259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6415 2022-09-27T16:09:51.2823125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:51.2823506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:51.2827278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:51.2827742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:51.3232489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:51.3232931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:51.3236040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:51.3236541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:51.4109031Z dist init r=0, world=2 2022-09-27T16:09:51.4512579Z dist init r=1, world=2 2022-09-27T16:09:51.4620939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:51.4622419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:51.4623576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:51.4723911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:51.4727398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:51.4728064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:51.7775497Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:09:51.7794690Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:09:51.7839697Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6442 2022-09-27T16:09:51.7872503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6443 2022-09-27T16:09:52.9011104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:52.9011548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:52.9014373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:52.9014787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:52.9308901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:52.9309399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:52.9312563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:52.9313097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:53.0284628Z dist init r=0, world=2 2022-09-27T16:09:53.0584389Z dist init r=1, world=2 2022-09-27T16:09:53.0693701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:53.0694298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:53.0695107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:53.0695626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:53.0801570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:53.0802035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:53.3905045Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:09:53.3924875Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:09:53.3969316Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6470 2022-09-27T16:09:53.4004254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6471 2022-09-27T16:09:54.4901571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:54.4901969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:54.4904650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:54.4905013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:54.5279082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:54.5279479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:54.5282466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:54.5282854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:54.6165355Z dist init r=0, world=2 2022-09-27T16:09:54.6528935Z dist init r=1, world=2 2022-09-27T16:09:54.6676570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:54.6677258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:54.6678216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:54.6678761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:54.6782749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:54.6783235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:55.0037556Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:09:55.0056779Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:09:55.0102105Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6498 2022-09-27T16:09:55.0137286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6499 2022-09-27T16:09:56.1043819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:56.1044208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:56.1046850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:56.1047319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:56.1047805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:56.1048212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:56.1051320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:56.1051675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:56.2329714Z dist init r=1, world=2 2022-09-27T16:09:56.2329951Z dist init r=0, world=2 2022-09-27T16:09:56.2540304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:56.2540760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:56.2541440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:56.2541958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:56.2646046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:56.5168045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:56.5168358Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:09:56.5186986Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:09:56.5232788Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6526 2022-09-27T16:09:56.5266551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6527 2022-09-27T16:09:57.6103952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:57.6104333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:57.6107168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:57.6107675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:57.6142271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:57.6142784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:57.6146674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:57.6147188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:57.7377814Z dist init r=1, world=2 2022-09-27T16:09:57.7420927Z dist init r=0, world=2 2022-09-27T16:09:57.7631042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:57.7631722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:57.7632388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:57.7632912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:57.7737666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:57.7738079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:58.0295867Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:09:58.0315811Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:09:58.0359624Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6554 2022-09-27T16:09:58.0392673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6555 2022-09-27T16:09:59.1438386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:59.1438767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:59.1441673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:59.1442027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:59.1741509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:09:59.1741854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:09:59.1745714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:09:59.1746096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:09:59.2700541Z dist init r=1, world=2 2022-09-27T16:09:59.3014447Z dist init r=0, world=2 2022-09-27T16:09:59.3224528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:09:59.3224955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:09:59.3225671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:59.3226496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:09:59.3230174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:09:59.3230620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:09:59.6426153Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:09:59.6444932Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:09:59.6490056Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6582 2022-09-27T16:09:59.6525264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6583 2022-09-27T16:10:00.7753050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:00.7753722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:00.7757104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:00.7757715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:00.7848577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:00.7849195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:00.7852470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:00.7853101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:00.9020344Z dist init r=1, world=2 2022-09-27T16:10:00.9119462Z dist init r=0, world=2 2022-09-27T16:10:00.9229978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:00.9230778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:00.9231962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:00.9232531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:00.9236640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:00.9237094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:01.2561650Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:01.2580341Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:01.2626197Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6610 2022-09-27T16:10:01.2661622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6611 2022-09-27T16:10:02.3915161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:02.3915783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:02.3918084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:02.3918694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:02.3919385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:02.3920018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:02.3924343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:02.3924965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:02.5178204Z dist init r=1, world=2 2022-09-27T16:10:02.5185941Z dist init r=0, world=2 2022-09-27T16:10:02.5397192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:02.5397731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:02.5398487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:02.5399238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:02.5403419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:02.5403953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:02.8693383Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:02.8712232Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:02.8757991Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6638 2022-09-27T16:10:02.8793107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6639 2022-09-27T16:10:04.0222831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:04.0223219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:04.0225249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:04.0225817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:04.0226703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:04.0227154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:04.0229921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:04.0230271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:04.1495733Z dist init r=0, world=2 2022-09-27T16:10:04.1496133Z dist init r=1, world=2 2022-09-27T16:10:04.1604185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:04.1605143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:04.1606218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:04.1705923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:04.1710745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:04.1711136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:04.4826419Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:04.4846403Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:04.4892202Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6666 2022-09-27T16:10:04.4925969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6667 2022-09-27T16:10:05.5825852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:05.5826234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:05.5827858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:05.5828455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:05.5829000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:05.5829362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:05.5832955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:05.5833299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:05.7085393Z dist init r=1, world=2 2022-09-27T16:10:05.7091943Z dist init r=0, world=2 2022-09-27T16:10:05.7303373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:05.7303783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:05.7304390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:05.7304927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:05.7409294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:05.7409660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:05.9957210Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:10:05.9975531Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:06.0020495Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6694 2022-09-27T16:10:06.0054184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6695 2022-09-27T16:10:07.1123340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:07.1123952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:07.1126645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:07.1127233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:07.1166753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:07.1167261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:07.1172085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:07.1172465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:07.2395860Z dist init r=1, world=2 2022-09-27T16:10:07.2440339Z dist init r=0, world=2 2022-09-27T16:10:07.2649493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:07.2650186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:07.2651076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:07.2651900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:07.2654787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:07.2655351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:07.6087482Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:07.6106834Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:07.6153077Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6722 2022-09-27T16:10:07.6187560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6723 2022-09-27T16:10:08.7173429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:08.7173798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:08.7176552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:08.7177104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:08.7202638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:08.7203173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:08.7206941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:08.7207308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:08.8445680Z dist init r=0, world=2 2022-09-27T16:10:08.8480853Z dist init r=1, world=2 2022-09-27T16:10:08.8655502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:08.8655921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:08.8656580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:08.8657156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:08.8660560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:08.8660920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:09.1220841Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:10:09.1239571Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:09.1278101Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6750 2022-09-27T16:10:09.1307910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6751 2022-09-27T16:10:10.2513615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:10.2514005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:10.2517269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:10.2517657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:10.2630358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:10.2631032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:10.2635188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:10.2635750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:10.3782994Z dist init r=1, world=2 2022-09-27T16:10:10.3892977Z dist init r=0, world=2 2022-09-27T16:10:10.4102510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:10.4103111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:10.4104122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:10.4104665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:10.4107662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:10.4108300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:10.7340128Z skip: Need at least 2 CUDA devices (1.612s) 2022-09-27T16:10:10.7359133Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:10.7403860Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6778 2022-09-27T16:10:10.7437654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6779 2022-09-27T16:10:11.8598345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:11.8598813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:11.8599265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:11.8599656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:11.8601526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:11.8601886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:11.8603468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:11.8603814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:11.9878488Z dist init r=1, world=2 2022-09-27T16:10:11.9878825Z dist init r=0, world=2 2022-09-27T16:10:12.0088633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:12.0089245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:12.0089934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:12.0090465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:12.0093570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:12.0094004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:12.3473190Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:12.3492219Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:12.3538252Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6806 2022-09-27T16:10:12.3572548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6807 2022-09-27T16:10:13.4584045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:13.4584431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:13.4587687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:13.4588032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:13.4604377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:13.4604740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:13.4608990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:13.4609359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:13.5850001Z dist init r=0, world=2 2022-09-27T16:10:13.5862824Z dist init r=1, world=2 2022-09-27T16:10:13.6058836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:13.6059301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:13.6059908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:13.6060447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:13.6165311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:13.6165880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:13.9608177Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:13.9626997Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:13.9672166Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6834 2022-09-27T16:10:13.9706725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6835 2022-09-27T16:10:15.0657583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:15.0657982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:15.0660851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:15.0661221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:15.0700270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:15.0700677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:15.0704381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:15.0704791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:15.1914179Z dist init r=1, world=2 2022-09-27T16:10:15.1956882Z dist init r=0, world=2 2022-09-27T16:10:15.2123828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:15.2124383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:15.2125041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:15.2125580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:15.2229910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:15.2230506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:15.4736869Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:10:15.4795334Z test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6862 2022-09-27T16:10:15.4822032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6863 2022-09-27T16:10:16.5969238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:16.5969838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:16.5972296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:16.5972912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:16.6013881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:16.6014505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:16.6018873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:16.6019495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:16.7234684Z dist init r=1, world=2 2022-09-27T16:10:16.7281490Z dist init r=0, world=2 2022-09-27T16:10:16.7443833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:16.7444255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:16.7444947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:16.7445488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:16.7449305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:16.7449693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:17.0855699Z skip: Need at least 2 CUDA devices (1.612s) 2022-09-27T16:10:17.0910691Z test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6890 2022-09-27T16:10:17.0945417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6891 2022-09-27T16:10:18.1964945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:18.1965579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:18.1967841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:18.1968463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:18.1991044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:18.1991588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:18.1996574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:18.1997191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:18.3221297Z dist init r=1, world=2 2022-09-27T16:10:18.3243207Z dist init r=0, world=2 2022-09-27T16:10:18.3453188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:18.3453887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:18.3454486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:18.3455153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:18.3459295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:18.3459688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:18.5976177Z skip: Need at least 2 CUDA devices (1.512s) 2022-09-27T16:10:18.6027868Z test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6918 2022-09-27T16:10:18.6061062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6919 2022-09-27T16:10:19.7198713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:19.7199089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:19.7202286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:19.7202656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:19.7457066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:19.7457453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:19.7460806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:19.7461198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:19.8470440Z dist init r=0, world=2 2022-09-27T16:10:19.8721508Z dist init r=1, world=2 2022-09-27T16:10:19.8830381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:19.8830927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:19.8831545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:19.8832062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:19.8937679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:19.8938111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:20.2096335Z skip: Need at least 2 CUDA devices (1.612s) 2022-09-27T16:10:20.2106583Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both (__main__.TestFSDPStateDict) 2022-09-27T16:10:20.2153916Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6946 2022-09-27T16:10:20.2187934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6947 2022-09-27T16:10:21.3451389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:21.3451777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:21.3454388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:21.3454809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:21.3675026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:21.3675395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:21.3678648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:21.3679019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:21.4729722Z dist init r=1, world=2 2022-09-27T16:10:21.4938368Z dist init r=0, world=2 2022-09-27T16:10:21.5148140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:21.5148868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:21.5149522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:21.5150064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:21.5153719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:21.5154106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:21.8219191Z skip: Need at least 2 CUDA devices (1.612s) 2022-09-27T16:10:21.8228387Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_first (__main__.TestFSDPStateDict) 2022-09-27T16:10:21.8274681Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6974 2022-09-27T16:10:21.8308771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6975 2022-09-27T16:10:22.9318551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:22.9318950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:22.9321829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:22.9322181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:22.9363747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:22.9364090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:22.9368073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:22.9368418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:23.0588471Z dist init r=0, world=2 2022-09-27T16:10:23.0638109Z dist init r=1, world=2 2022-09-27T16:10:23.0799155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:23.0799756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:23.0800434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:23.0800966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:23.0804456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:23.0804823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:23.3338491Z skip: Need at least 2 CUDA devices (1.512s) 2022-09-27T16:10:23.3347969Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_second (__main__.TestFSDPStateDict) 2022-09-27T16:10:23.3393553Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7002 2022-09-27T16:10:23.3427842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7003 2022-09-27T16:10:24.4453318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:24.4453996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:24.4454780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:24.4455321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:24.4473897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:24.4475079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:24.4476014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:24.4476707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:24.5744290Z dist init r=1, world=2 2022-09-27T16:10:24.5747698Z dist init r=0, world=2 2022-09-27T16:10:24.5957852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:24.5958362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:24.5959046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:24.5959584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:24.5964865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:24.5965246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:24.8461324Z skip: Need at least 2 CUDA devices (1.512s) 2022-09-27T16:10:24.8471066Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both (__main__.TestFSDPStateDict) 2022-09-27T16:10:24.8516556Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7030 2022-09-27T16:10:24.8549955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7031 2022-09-27T16:10:25.9574923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:25.9575319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:25.9578647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:25.9579087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:25.9682068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:25.9682544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:25.9686329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:25.9686848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:26.0846149Z dist init r=1, world=2 2022-09-27T16:10:26.0954942Z dist init r=0, world=2 2022-09-27T16:10:26.1155703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:26.1156131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:26.1156749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:26.1157287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:26.1263257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:26.1263843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:26.4584312Z skip: Need at least 2 CUDA devices (1.612s) 2022-09-27T16:10:26.4595018Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_first (__main__.TestFSDPStateDict) 2022-09-27T16:10:26.4641426Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7058 2022-09-27T16:10:26.4675236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7059 2022-09-27T16:10:27.5887825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:27.5888515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:27.5891744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:27.5892320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:27.6034342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:27.6035095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:27.6039132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:27.6039766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:27.7191042Z dist init r=0, world=2 2022-09-27T16:10:27.7310927Z dist init r=1, world=2 2022-09-27T16:10:27.7419245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:27.7419632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:27.7420248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:27.7420781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:27.7525118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:27.7525699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:28.0707423Z skip: Need at least 2 CUDA devices (1.612s) 2022-09-27T16:10:28.0717938Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_second (__main__.TestFSDPStateDict) 2022-09-27T16:10:28.0763582Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7086 2022-09-27T16:10:28.0797970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7087 2022-09-27T16:10:29.2040687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:29.2043559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:29.2044164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:29.2044523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:29.2109428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:29.2109792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:29.2113514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:29.2113880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:29.3331672Z dist init r=1, world=2 2022-09-27T16:10:29.3383319Z dist init r=0, world=2 2022-09-27T16:10:29.3540935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:29.3541483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:29.3542251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:29.3542865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:29.3547343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:29.3549178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:29.6830512Z skip: Need at least 2 CUDA devices (1.612s) 2022-09-27T16:10:29.6852146Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:29.6897700Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7114 2022-09-27T16:10:29.6931033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7115 2022-09-27T16:10:30.8031732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:30.8032134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:30.8035202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:30.8035699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:30.8121881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:30.8122442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:30.8126348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:30.8126903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:30.9300853Z dist init r=0, world=2 2022-09-27T16:10:30.9377770Z dist init r=1, world=2 2022-09-27T16:10:30.9486156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:30.9486904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:30.9487574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:30.9488107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:30.9591841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:30.9592196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:31.1964259Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:10:31.1984770Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:31.2028409Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7142 2022-09-27T16:10:31.2061401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7143 2022-09-27T16:10:32.3030533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:32.3030997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:32.3033968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:32.3034481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:32.3048430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:32.3049024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:32.3053038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:32.3053519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:32.4304369Z dist init r=0, world=2 2022-09-27T16:10:32.4313524Z dist init r=1, world=2 2022-09-27T16:10:32.4514679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:32.4515221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:32.4516014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:32.4516581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:32.4520487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:32.4521099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:32.7091172Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:10:32.7111973Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:32.7158903Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7170 2022-09-27T16:10:32.7193512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7171 2022-09-27T16:10:33.8288186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:33.8288565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:33.8291520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:33.8292214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:33.8584482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:33.8585018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:33.8588081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:33.8588593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:33.9543725Z dist init r=1, world=2 2022-09-27T16:10:33.9819107Z dist init r=0, world=2 2022-09-27T16:10:33.9953483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:33.9954163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:33.9955077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:33.9955619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:33.9959603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:33.9960042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:34.3226414Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:34.3247381Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:34.3292592Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7198 2022-09-27T16:10:34.3326929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7199 2022-09-27T16:10:35.4279764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:35.4280148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:35.4283029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:35.4283376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:35.4671794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:35.4672151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:35.4675806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:35.4676227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:35.5551813Z dist init r=1, world=2 2022-09-27T16:10:35.5939875Z dist init r=0, world=2 2022-09-27T16:10:35.6063197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:35.6063764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:35.6064388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:35.6065160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:35.6068742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:35.6069248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:35.9358901Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:35.9379422Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:35.9424572Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7226 2022-09-27T16:10:35.9458534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7227 2022-09-27T16:10:37.0450467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:37.0450995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:37.0452398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:37.0452924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:37.0587755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:37.0588270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:37.0591857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:37.0592436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:37.1711699Z dist init r=1, world=2 2022-09-27T16:10:37.1852564Z dist init r=0, world=2 2022-09-27T16:10:37.2021426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:37.2022042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:37.2022614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:37.2023144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:37.2127373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:37.2127937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:37.5490717Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:37.5511181Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:37.5556948Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7254 2022-09-27T16:10:37.5590772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7255 2022-09-27T16:10:38.6665233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:38.6665606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:38.6666125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:38.6666459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:38.6669043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:38.6669387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:38.6669835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:38.6670180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:38.7926736Z dist init r=1, world=2 2022-09-27T16:10:38.7934059Z dist init r=0, world=2 2022-09-27T16:10:38.8136042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:38.8136717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:38.8137345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:38.8137931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:38.8142025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:38.8143109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:39.0624155Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:10:39.0645143Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:39.0689249Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7282 2022-09-27T16:10:39.0723078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7283 2022-09-27T16:10:40.1675889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:40.1676544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:40.1678657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:40.1679266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:40.1743933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:40.1744594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:40.1748729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:40.1749357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:40.2935825Z dist init r=1, world=2 2022-09-27T16:10:40.2996788Z dist init r=0, world=2 2022-09-27T16:10:40.3208096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:40.3208678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:40.3209365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:40.3209893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:40.3213936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:40.3214508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:40.5754151Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:10:40.5776060Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:40.5821219Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7310 2022-09-27T16:10:40.5854273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7311 2022-09-27T16:10:41.6943256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:41.6943659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:41.6946173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:41.6946530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:41.6999166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:41.6999586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:41.7003710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:41.7004062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:41.8211028Z dist init r=1, world=2 2022-09-27T16:10:41.8270216Z dist init r=0, world=2 2022-09-27T16:10:41.8479935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:41.8480646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:41.8481268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:41.8482097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:41.8485588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:41.8486229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:42.0884668Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:10:42.0905586Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:42.0950192Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7338 2022-09-27T16:10:42.0982587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7339 2022-09-27T16:10:43.2105376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:43.2105752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:43.2109110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:43.2109613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:43.2214494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:43.2214992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:43.2219042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:43.2219430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:43.3368967Z dist init r=0, world=2 2022-09-27T16:10:43.3479762Z dist init r=1, world=2 2022-09-27T16:10:43.3588113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:43.3588519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:43.3589124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:43.3589659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:43.3694933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:43.3695999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:43.7016279Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:43.7037436Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:43.7083181Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7366 2022-09-27T16:10:43.7118649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7367 2022-09-27T16:10:44.8200389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:44.8201032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:44.8204132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:44.8204697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:44.8565365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:44.8565983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:44.8568943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:44.8569542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:44.9469575Z dist init r=0, world=2 2022-09-27T16:10:44.9822140Z dist init r=1, world=2 2022-09-27T16:10:44.9930427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:44.9931151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:44.9931806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:44.9932323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:45.0036954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:45.0037426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:45.3224745Z skip: Need at least 2 CUDA devices (1.621s) 2022-09-27T16:10:45.3245780Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:45.3290640Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7394 2022-09-27T16:10:45.3325711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7395 2022-09-27T16:10:46.4207786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:46.4208436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:46.4211645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:46.4212269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:46.4574280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:46.4574915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:46.4578038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:46.4578432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:46.5467949Z dist init r=0, world=2 2022-09-27T16:10:46.5835781Z dist init r=1, world=2 2022-09-27T16:10:46.5943905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:46.5944394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:46.5945287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:46.5945817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:46.6050821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:46.6051392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:46.9357994Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:46.9379230Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:46.9424918Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7422 2022-09-27T16:10:46.9458824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7423 2022-09-27T16:10:48.0399796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:48.0400489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:48.0407002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:48.0407604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:48.0810503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:48.0810941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:48.0813315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:48.0813671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:48.1686576Z dist init r=1, world=2 2022-09-27T16:10:48.2100149Z dist init r=0, world=2 2022-09-27T16:10:48.2299571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:48.2300225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:48.2300813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:48.2301343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:48.2304999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:48.2305590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:48.5491109Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:48.5515322Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:48.5560487Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7450 2022-09-27T16:10:48.5594425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7451 2022-09-27T16:10:49.6483657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:49.6484056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:49.6486802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:49.6487300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:49.6496824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:49.6497501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:49.6502246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:49.6502814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:49.7755358Z dist init r=0, world=2 2022-09-27T16:10:49.7766028Z dist init r=1, world=2 2022-09-27T16:10:49.7963923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:49.7964421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:49.7965149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:49.7965682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:49.8070266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:49.8070883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:50.0625162Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:10:50.0648120Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:50.0692467Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7478 2022-09-27T16:10:50.0725092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7479 2022-09-27T16:10:51.1656077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:51.1656618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:51.1661123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:51.1661708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:51.1894000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:51.1894648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:51.1897439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:51.1898098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:51.2932314Z dist init r=1, world=2 2022-09-27T16:10:51.3187797Z dist init r=0, world=2 2022-09-27T16:10:51.3343332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:51.3343857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:51.3344484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:51.3345153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:51.3349922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:51.3350411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:51.6757667Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:51.6780709Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:51.6827485Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7506 2022-09-27T16:10:51.6863500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7507 2022-09-27T16:10:52.7896837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:52.7897863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:52.7898673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:52.7899231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:52.7900286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:52.7900888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:52.7901694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:52.7902255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:52.9162636Z dist init r=1, world=2 2022-09-27T16:10:52.9163031Z dist init r=0, world=2 2022-09-27T16:10:52.9370981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:52.9371528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:52.9372143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:52.9372875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:52.9377254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:52.9377644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:53.1897214Z skip: Need at least 2 CUDA devices (1.514s) 2022-09-27T16:10:53.1921072Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:53.1964759Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7534 2022-09-27T16:10:53.1996698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7535 2022-09-27T16:10:54.3014965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:54.3015472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:54.3016460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:54.3017028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:54.3017973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:54.3018328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:54.3021341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:54.3021698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:54.4281772Z dist init r=1, world=2 2022-09-27T16:10:54.4306607Z dist init r=0, world=2 2022-09-27T16:10:54.4490437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:54.4490940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:54.4491591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:54.4492344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:54.4496695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:54.4497437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:54.7030776Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:10:54.7053844Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:54.7098662Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7562 2022-09-27T16:10:54.7131966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7563 2022-09-27T16:10:55.8178422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:55.8179097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:55.8182014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:55.8182578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:55.8366928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:55.8367575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:55.8371734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:55.8372349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:55.9439588Z dist init r=0, world=2 2022-09-27T16:10:55.9616612Z dist init r=1, world=2 2022-09-27T16:10:55.9749333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:55.9749803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:55.9750443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:55.9751052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:55.9855271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:55.9855758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:56.3166125Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:10:56.3188900Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:56.3235488Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7590 2022-09-27T16:10:56.3269427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7591 2022-09-27T16:10:57.4626747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:57.4627392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:57.4631074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:57.4631623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:57.4632363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:57.4632924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:57.4636008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:57.4636596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:57.5911995Z dist init r=0, world=2 2022-09-27T16:10:57.5912330Z dist init r=1, world=2 2022-09-27T16:10:57.6121563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:57.6122129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:57.6122749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:57.6123268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:57.6228495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:57.6229113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:57.9307138Z skip: Need at least 2 CUDA devices (1.614s) 2022-09-27T16:10:57.9330053Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict) 2022-09-27T16:10:57.9376104Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7618 2022-09-27T16:10:57.9409044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7619 2022-09-27T16:10:59.0433812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:59.0434462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:59.0438879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:59.0439515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:59.0513011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:10:59.0513644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:10:59.0517343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:10:59.1712971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:10:59.1713431Z dist init r=1, world=2 2022-09-27T16:10:59.1780304Z dist init r=0, world=2 2022-09-27T16:10:59.1989257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:10:59.1989748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:10:59.1990423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:59.1991050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:10:59.1995576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:10:59.1995934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:10:59.4440449Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:10:59.4463215Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict) 2022-09-27T16:10:59.4506989Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7646 2022-09-27T16:10:59.4540627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7647 2022-09-27T16:11:00.5593305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:00.5593943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:00.5597684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:00.5598295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:00.5684688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:00.5685297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:00.5688737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:00.5689363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:00.6858804Z dist init r=0, world=2 2022-09-27T16:11:00.6940777Z dist init r=1, world=2 2022-09-27T16:11:00.7048969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:00.7049429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:00.7050131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:00.7050651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:00.7155452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:00.7156023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:00.9570987Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:11:00.9588075Z test_state_dict_rank0_offload_save_load_flow (__main__.TestFSDPStateDict) 2022-09-27T16:11:00.9632455Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7674 2022-09-27T16:11:00.9665041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7675 2022-09-27T16:11:02.0756868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:02.0757518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:02.0760147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:02.0760526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:02.0842135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:02.0842617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:02.0846543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:02.0846937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:02.2017879Z dist init r=1, world=2 2022-09-27T16:11:02.2102788Z dist init r=0, world=2 2022-09-27T16:11:02.2312943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:02.2313396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:02.2314183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:02.2314923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:02.2319823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:02.2320517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:02.5698131Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:11:02.5765560Z test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7702 2022-09-27T16:11:02.5816147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7703 2022-09-27T16:11:03.6786061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:03.6786442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:03.6788620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:03.6788964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:03.6958831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:03.6959180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:03.6962798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:03.6963147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:03.8038443Z dist init r=1, world=2 2022-09-27T16:11:03.8209794Z dist init r=0, world=2 2022-09-27T16:11:03.8346833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:03.8347333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:03.8347946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:03.8348558Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:03.8352759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:03.8353159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:04.0846591Z skip: Need at least 2 CUDA devices (1.515s) 2022-09-27T16:11:04.0896696Z test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7730 2022-09-27T16:11:04.0929789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7731 2022-09-27T16:11:05.1855530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:05.1855926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:05.1858736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:05.1859302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:05.1906121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:05.1906604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:05.1910997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:05.1911625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:05.3120178Z dist init r=0, world=2 2022-09-27T16:11:05.3159098Z dist init r=1, world=2 2022-09-27T16:11:05.3267141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:05.3267581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:05.3268490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:05.3269026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:05.3373411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:05.3373781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:05.5960455Z skip: Need at least 2 CUDA devices (1.511s) 2022-09-27T16:11:05.6009790Z test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7758 2022-09-27T16:11:05.6043522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7759 2022-09-27T16:11:06.7032315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:06.7032716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:06.7036174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:06.7036532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:06.7253759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:06.7254104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:06.7257706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:06.7258048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:06.8296433Z dist init r=0, world=2 2022-09-27T16:11:06.8518614Z dist init r=1, world=2 2022-09-27T16:11:06.8708811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:06.8709319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:06.8709999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:06.8710511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:06.8714361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:06.8715176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:07.1074226Z skip: Need at least 2 CUDA devices (1.511s) 2022-09-27T16:11:07.1143158Z test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7786 2022-09-27T16:11:07.1175648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7787 2022-09-27T16:11:08.2176794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:08.2177429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:08.2180226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:08.2180848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:08.2454325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:08.2454976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:08.2458788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:08.2459669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:08.3441081Z dist init r=1, world=2 2022-09-27T16:11:08.3710177Z dist init r=0, world=2 2022-09-27T16:11:08.3850748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:08.3851301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:08.3852102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:08.3852660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:08.3856627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:08.3857066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:08.7208434Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:11:08.7279372Z test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7814 2022-09-27T16:11:08.7313726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7815 2022-09-27T16:11:09.8430606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:09.8431347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:09.8434922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:09.8435523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:09.8582126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:09.8582735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:09.8586717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:09.8587341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:09.9684426Z dist init r=1, world=2 2022-09-27T16:11:09.9835417Z dist init r=0, world=2 2022-09-27T16:11:09.9994993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:09.9995697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:09.9996333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:09.9996867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:10.0100664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:10.0101223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:10.3346038Z skip: Need at least 2 CUDA devices (1.614s) 2022-09-27T16:11:10.3415119Z test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7842 2022-09-27T16:11:10.3448697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7843 2022-09-27T16:11:11.4386157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:11.4386540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:11.4389391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:11.4389760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:11.4492279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:11.4492765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:11.4496170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:11.4496522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:11.5644831Z dist init r=1, world=2 2022-09-27T16:11:11.5741430Z dist init r=0, world=2 2022-09-27T16:11:11.5951511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:11.5952228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:11.5953292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:11.5953850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:11.5957239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:11.5957890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:11.8483160Z skip: Need at least 2 CUDA devices (1.514s) 2022-09-27T16:11:11.8532130Z test_state_dict_type (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7870 2022-09-27T16:11:11.8563953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7871 2022-09-27T16:11:12.9570711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:12.9571106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:12.9573665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:12.9574152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:12.9743027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:12.9743488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:12.9746979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:12.9747324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:13.0834087Z dist init r=1, world=2 2022-09-27T16:11:13.0999081Z dist init r=0, world=2 2022-09-27T16:11:13.1207839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:13.1208523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:13.1209310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:13.1209848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:13.1213254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:13.1213802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:13.3594397Z skip: Need at least 2 CUDA devices (1.511s) 2022-09-27T16:11:13.3660053Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7898 2022-09-27T16:11:13.3692363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7899 2022-09-27T16:11:14.4655765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:14.4656457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:14.4657255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:14.4657798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:14.4658770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:14.4659369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:14.4660080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:14.4660665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:14.5922860Z dist init r=0, world=2 2022-09-27T16:11:14.5925429Z dist init r=1, world=2 2022-09-27T16:11:14.6034009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:14.6034796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:14.6035414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:14.6035940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:14.6140292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:14.6140705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:14.8722503Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:11:14.8788708Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7926 2022-09-27T16:11:14.8821759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7927 2022-09-27T16:11:15.9710781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:15.9711380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:15.9712093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:15.9712662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:15.9713871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:15.9714427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:15.9715376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:15.9715961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:16.0988995Z dist init r=0, world=2 2022-09-27T16:11:16.0989327Z dist init r=1, world=2 2022-09-27T16:11:16.1097571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:16.1099395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:16.1100476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:16.1199484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:16.1204246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:16.1204861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:16.3852849Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:11:16.3920063Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7954 2022-09-27T16:11:16.3953635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7955 2022-09-27T16:11:17.4999209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:17.5002096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:17.5002572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:17.5002926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:17.5118268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:17.5118596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:17.5122366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:17.5122721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:17.6261197Z dist init r=1, world=2 2022-09-27T16:11:17.6377913Z dist init r=0, world=2 2022-09-27T16:11:17.6588103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:17.6588644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:17.6589255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:17.6589993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:17.6594015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:17.6594862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:17.9985454Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:11:18.0055153Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7982 2022-09-27T16:11:18.0089406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7983 2022-09-27T16:11:19.0985180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:19.0985791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:19.0988547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:19.0989167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:19.1041054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:19.1041615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:19.1045943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:19.1046531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:19.2247284Z dist init r=1, world=2 2022-09-27T16:11:19.2307597Z dist init r=0, world=2 2022-09-27T16:11:19.2516962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:19.2517663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:19.2518284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:19.2518881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:19.2623020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:19.2623604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:19.5118683Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:11:19.5184247Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8010 2022-09-27T16:11:19.5217534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8011 2022-09-27T16:11:20.6099203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:20.6099638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:20.6102871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:20.6103251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:20.6219843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:20.6220201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:20.6224218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:20.6224611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:20.7359479Z dist init r=1, world=2 2022-09-27T16:11:20.7476508Z dist init r=0, world=2 2022-09-27T16:11:20.7669643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:20.7670329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:20.7671020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:20.7671560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:20.7675421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:20.7676080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:21.0247049Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:11:21.0312724Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8038 2022-09-27T16:11:21.0343864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8039 2022-09-27T16:11:22.1378232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:22.1378875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:22.1381341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:22.1381777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:22.1435266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:22.1435605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:22.1439924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:22.1440287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:22.2641969Z dist init r=1, world=2 2022-09-27T16:11:22.2683759Z dist init r=0, world=2 2022-09-27T16:11:22.2850323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:22.2851041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:22.2851942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:22.2852676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:22.2856781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:22.2857463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:22.5375028Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:11:22.5441844Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8066 2022-09-27T16:11:22.5474302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8067 2022-09-27T16:11:23.6430830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:23.6431499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:23.6435099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:23.6435725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:23.6442575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:23.6443199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:23.6446970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:23.6447398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:23.7699524Z dist init r=1, world=2 2022-09-27T16:11:23.7702608Z dist init r=0, world=2 2022-09-27T16:11:23.7911917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:23.7912590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:23.7913558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:23.7914327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:23.7918806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:23.7919203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:24.0508329Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:11:24.0576697Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8094 2022-09-27T16:11:24.0610070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8095 2022-09-27T16:11:25.1677758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:25.1678183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:25.1681318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:25.1681669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:25.1689512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:25.1689875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:25.1694073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:25.1694427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:25.2975307Z dist init r=1, world=2 2022-09-27T16:11:25.2980063Z dist init r=0, world=2 2022-09-27T16:11:25.3184640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:25.3185131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:25.3185936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:25.3186481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:25.3189898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:25.3190510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:25.6645891Z skip: Need at least 2 CUDA devices (1.614s) 2022-09-27T16:11:25.6696071Z test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8122 2022-09-27T16:11:25.6729722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8123 2022-09-27T16:11:26.7744930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:26.7745423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:26.7748038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:26.7748551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:26.8058188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:26.8058593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:26.8061796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:26.8062154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:26.9020171Z dist init r=1, world=2 2022-09-27T16:11:26.9320955Z dist init r=0, world=2 2022-09-27T16:11:26.9530800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:11:26.9531466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:11:26.9532119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:26.9532929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:11:26.9536668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:26.9537044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:27.2762615Z skip: Need at least 2 CUDA devices (1.611s) 2022-09-27T16:11:27.2762983Z 2022-09-27T16:11:27.2763450Z ---------------------------------------------------------------------- 2022-09-27T16:11:27.2763701Z Ran 70 tests in 109.657s 2022-09-27T16:11:27.2763817Z 2022-09-27T16:11:27.2764064Z OK (skipped=70) 2022-09-27T16:11:27.2764161Z 2022-09-27T16:11:27.2764247Z Generating XML reports... 2022-09-27T16:11:27.2875109Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20220927160937.xml 2022-09-27T16:11:27.6015403Z Running distributed/test_c10d_nccl ... [2022-09-27 16:11:27.601161] 2022-09-27T16:11:27.6015984Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:11:27.601270] 2022-09-27T16:11:28.8327733Z , <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_nncl_rank_membership>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]> 2022-09-27T16:11:28.8330179Z test_all_reduce_coalesced_nccl (__main__.CommTest) 2022-09-27T16:11:28.8330574Z test_broadcast_coalesced_nccl (__main__.CommTest) 2022-09-27T16:11:28.8330932Z test_nccl_barrier (__main__.CommTest) 2022-09-27T16:11:28.8331294Z test_nccl_barrier_device_ids (__main__.CommTest) 2022-09-27T16:11:28.8331620Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) 2022-09-27T16:11:28.8332055Z test_nccl_barrier_timeout (__main__.CommTest) 2022-09-27T16:11:28.8332477Z test_nccl_barrier_timeout_new_group (__main__.CommTest) 2022-09-27T16:11:28.8332960Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) 2022-09-27T16:11:28.8333268Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) 2022-09-27T16:11:28.8333538Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) 2022-09-27T16:11:28.8333829Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) 2022-09-27T16:11:28.8334069Z test_nncl_rank_membership (__main__.CommTest) 2022-09-27T16:11:28.8334341Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) 2022-09-27T16:11:28.8334649Z test_sequence_num_incremented_nccl_default (__main__.CommTest) 2022-09-27T16:11:28.8334918Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) 2022-09-27T16:11:28.8335240Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) 2022-09-27T16:11:28.8335480Z test_sequence_num_set_nccl_new_group (__main__.CommTest) 2022-09-27T16:11:28.8335719Z test_tensor_dtype_complex (__main__.CommTest) 2022-09-27T16:11:28.8336005Z test_tensor_dtype_mismatch (__main__.CommTest) 2022-09-27T16:11:28.8336744Z , <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_reduce_scatter_work_wait_gpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]> 2022-09-27T16:11:28.8337647Z test_allgather_work_wait_gpu (__main__.CompilerTest) 2022-09-27T16:11:28.8337899Z test_allreduce_work_wait_gpu (__main__.CompilerTest) 2022-09-27T16:11:28.8338249Z test_broadcast_work_wait_gpu (__main__.CompilerTest) 2022-09-27T16:11:28.8338527Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) 2022-09-27T16:11:28.8338779Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) 2022-09-27T16:11:28.8339092Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) 2022-09-27T16:11:28.8339350Z test_scatter_work_wait_gpu (__main__.CompilerTest) 2022-09-27T16:11:28.8347128Z , <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_channels_last_contig>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]> 2022-09-27T16:11:28.8354941Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8355363Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8355720Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8356126Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8356454Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8356815Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8357144Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8357466Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8357849Z test_channels_last_contig (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8358173Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8358572Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8358918Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8359323Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8359696Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8360183Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8360557Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8361014Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8361372Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8361778Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8362161Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8362583Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8362945Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8363351Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8363697Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8364051Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8364395Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8364795Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8365126Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8365437Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8365732Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8366048Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8366373Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8366683Z test_failure_recovery (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8367059Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8367412Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8367744Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8368106Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8368481Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8368851Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8369157Z test_fp16 (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8369449Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8369769Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8370063Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8370396Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8370800Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8371112Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8371422Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8371775Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8372138Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8372493Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8372833Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8373200Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8373534Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8373909Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8374271Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8374637Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8374971Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8375270Z test_no_grad (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8375599Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8375903Z test_pass_default_pg (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8376196Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8376534Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8376865Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8377190Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) 2022-09-27T16:11:28.8377486Z 2022-09-27T16:11:28.8378494Z , <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]> 2022-09-27T16:11:28.8379445Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) 2022-09-27T16:11:28.8379752Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) 2022-09-27T16:11:28.8380050Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) 2022-09-27T16:11:28.8380335Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) 2022-09-27T16:11:28.8380634Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) 2022-09-27T16:11:28.8380931Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) 2022-09-27T16:11:28.8381211Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) 2022-09-27T16:11:28.8381502Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) 2022-09-27T16:11:28.8381772Z test_nccl_timeout (__main__.NcclErrorHandlingTest) 2022-09-27T16:11:28.8382096Z ]> 2022-09-27T16:11:28.8382431Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) 2022-09-27T16:11:28.8383959Z , <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_send_recv>]> 2022-09-27T16:11:28.8385532Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8385803Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8386075Z test_allgather_ops (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8386341Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8386602Z test_barrier (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8386844Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8387104Z test_empty_tensors (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8387364Z test_gather_checks (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8387611Z test_gather_ops (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8387866Z test_gather_stress (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8388125Z test_reduce_ops (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8388387Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8388674Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8388955Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8389212Z test_scatter_checks (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8389473Z test_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8389731Z test_scatter_stress (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8389989Z test_send_recv (__main__.ProcessGroupNCCLTest) 2022-09-27T16:11:28.8390287Z ]> 2022-09-27T16:11:28.8390592Z test_common_errors (__main__.RendezvousEnvTest) 2022-09-27T16:11:28.8390977Z 2022-09-27T16:11:28.8391280Z ]> 2022-09-27T16:11:28.8391598Z test_default_store_timeout_nccl (__main__.TimeoutTest) 2022-09-27T16:11:29.9043835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:29.9044199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:29.9046881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:29.9047267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:30.0399954Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:30.0409401Z 2022-09-27T16:11:30.0409481Z Running tests... 2022-09-27T16:11:30.0410028Z ---------------------------------------------------------------------- 2022-09-27T16:11:30.0419040Z test_all_reduce_coalesced_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:11:30.0419374Z 2022-09-27T16:11:30.0419750Z ---------------------------------------------------------------------- 2022-09-27T16:11:30.0420157Z Ran 1 test in 0.001s 2022-09-27T16:11:30.0420350Z 2022-09-27T16:11:30.0420423Z OK (skipped=1) 2022-09-27T16:11:30.0420532Z 2022-09-27T16:11:30.0420616Z Generating XML reports... 2022-09-27T16:11:30.0443940Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161130.xml 2022-09-27T16:11:31.3278872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:31.3279245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:31.3282341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:31.3282967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:31.4660668Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:31.4670070Z 2022-09-27T16:11:31.4670206Z Running tests... 2022-09-27T16:11:31.4670746Z ---------------------------------------------------------------------- 2022-09-27T16:11:31.4677362Z test_broadcast_coalesced_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:11:31.4677715Z 2022-09-27T16:11:31.4678126Z ---------------------------------------------------------------------- 2022-09-27T16:11:31.4678553Z Ran 1 test in 0.001s 2022-09-27T16:11:31.4678675Z 2022-09-27T16:11:31.4678747Z OK (skipped=1) 2022-09-27T16:11:31.4678858Z 2022-09-27T16:11:31.4678942Z Generating XML reports... 2022-09-27T16:11:31.4702062Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161131.xml 2022-09-27T16:11:32.7645013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:32.7645399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:32.7647934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:32.7648321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:32.9003968Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:32.9013170Z 2022-09-27T16:11:32.9013403Z Running tests... 2022-09-27T16:11:32.9014021Z ---------------------------------------------------------------------- 2022-09-27T16:11:32.9030304Z test_nccl_barrier (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:11:32.9030673Z 2022-09-27T16:11:32.9031097Z ---------------------------------------------------------------------- 2022-09-27T16:11:32.9031491Z Ran 1 test in 0.002s 2022-09-27T16:11:32.9031605Z 2022-09-27T16:11:32.9031676Z OK (skipped=1) 2022-09-27T16:11:32.9031782Z 2022-09-27T16:11:32.9031862Z Generating XML reports... 2022-09-27T16:11:32.9055406Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161132.xml 2022-09-27T16:11:34.1843565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:34.1843992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:34.1845880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:34.1846249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:34.3223573Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:34.3232497Z 2022-09-27T16:11:34.3232651Z Running tests... 2022-09-27T16:11:34.3233048Z ---------------------------------------------------------------------- 2022-09-27T16:11:34.3238829Z test_nccl_barrier_device_ids (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:11:34.3239195Z 2022-09-27T16:11:34.3239459Z ---------------------------------------------------------------------- 2022-09-27T16:11:34.3239708Z Ran 1 test in 0.001s 2022-09-27T16:11:34.3239823Z 2022-09-27T16:11:34.3239894Z OK (skipped=1) 2022-09-27T16:11:34.3240002Z 2022-09-27T16:11:34.3240074Z Generating XML reports... 2022-09-27T16:11:34.3263842Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161134.xml 2022-09-27T16:11:35.5972926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:35.5973542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:35.5975507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:35.5976072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:35.7335845Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:35.7345819Z 2022-09-27T16:11:35.7346136Z Running tests... 2022-09-27T16:11:35.7357681Z ---------------------------------------------------------------------- 2022-09-27T16:11:35.7358419Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:11:35.7358809Z 2022-09-27T16:11:35.7359152Z ---------------------------------------------------------------------- 2022-09-27T16:11:35.7359546Z Ran 1 test in 0.001s 2022-09-27T16:11:35.7359732Z 2022-09-27T16:11:35.7359846Z OK (skipped=1) 2022-09-27T16:11:35.7360028Z 2022-09-27T16:11:35.7360169Z Generating XML reports... 2022-09-27T16:11:35.7382252Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161135.xml 2022-09-27T16:11:37.0071894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:37.0072517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:37.0075184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:37.0075775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:37.1440275Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:37.1450168Z 2022-09-27T16:11:37.1450482Z Running tests... 2022-09-27T16:11:37.1451121Z ---------------------------------------------------------------------- 2022-09-27T16:11:37.1458172Z test_nccl_barrier_timeout (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:11:37.1458467Z 2022-09-27T16:11:37.1458858Z ---------------------------------------------------------------------- 2022-09-27T16:11:37.1459316Z Ran 1 test in 0.001s 2022-09-27T16:11:37.1459539Z 2022-09-27T16:11:37.1459638Z OK (skipped=1) 2022-09-27T16:11:37.1459738Z 2022-09-27T16:11:37.1459830Z Generating XML reports... 2022-09-27T16:11:37.1482801Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161137.xml 2022-09-27T16:11:38.4273778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:38.4274433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:38.4277542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:38.4278136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:38.5652685Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:38.5662349Z 2022-09-27T16:11:38.5662785Z Running tests... 2022-09-27T16:11:38.5663202Z ---------------------------------------------------------------------- 2022-09-27T16:11:38.5671508Z test_nccl_barrier_timeout_new_group (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:11:38.5672254Z 2022-09-27T16:11:38.5672595Z ---------------------------------------------------------------------- 2022-09-27T16:11:38.5672925Z Ran 1 test in 0.001s 2022-09-27T16:11:38.5673042Z 2022-09-27T16:11:38.5673117Z OK (skipped=1) 2022-09-27T16:11:38.5673224Z 2022-09-27T16:11:38.5673310Z Generating XML reports... 2022-09-27T16:11:38.5696414Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161138.xml 2022-09-27T16:11:39.8350997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:39.8351634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:39.8354193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:39.8355044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:39.9696072Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:39.9704763Z 2022-09-27T16:11:39.9704892Z Running tests... 2022-09-27T16:11:39.9705468Z ---------------------------------------------------------------------- 2022-09-27T16:11:39.9714432Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:11:39.9715182Z 2022-09-27T16:11:39.9715710Z ---------------------------------------------------------------------- 2022-09-27T16:11:39.9716171Z Ran 1 test in 0.001s 2022-09-27T16:11:39.9716380Z 2022-09-27T16:11:39.9716459Z OK (skipped=1) 2022-09-27T16:11:39.9716582Z 2022-09-27T16:11:39.9716670Z Generating XML reports... 2022-09-27T16:11:39.9738736Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161139.xml 2022-09-27T16:11:41.2434752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:41.2435147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:41.2437738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:41.2438132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:41.3782123Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:41.3791443Z 2022-09-27T16:11:41.3791781Z Running tests... 2022-09-27T16:11:41.3792165Z ---------------------------------------------------------------------- 2022-09-27T16:11:41.3796281Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:11:41.3797077Z 2022-09-27T16:11:41.3797398Z ---------------------------------------------------------------------- 2022-09-27T16:11:41.3797825Z Ran 1 test in 0.001s 2022-09-27T16:11:41.3798003Z 2022-09-27T16:11:41.3798135Z OK (skipped=1) 2022-09-27T16:11:41.3798337Z 2022-09-27T16:11:41.3798495Z Generating XML reports... 2022-09-27T16:11:41.3822377Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161141.xml 2022-09-27T16:11:42.6640763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:42.6641185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:42.6642923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:42.6643300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:42.8015341Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:42.8024741Z 2022-09-27T16:11:42.8024850Z Running tests... 2022-09-27T16:11:42.8025726Z ---------------------------------------------------------------------- 2022-09-27T16:11:42.8029939Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:11:42.8030460Z 2022-09-27T16:11:42.8030775Z ---------------------------------------------------------------------- 2022-09-27T16:11:42.8031065Z Ran 1 test in 0.000s 2022-09-27T16:11:42.8031181Z 2022-09-27T16:11:42.8031254Z OK (skipped=1) 2022-09-27T16:11:42.8031366Z 2022-09-27T16:11:42.8031439Z Generating XML reports... 2022-09-27T16:11:42.8055617Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161142.xml 2022-09-27T16:11:44.0954238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:44.0954883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:44.0956731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:44.0957096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:44.2327717Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:44.2337158Z 2022-09-27T16:11:44.2337354Z Running tests... 2022-09-27T16:11:44.2337927Z ---------------------------------------------------------------------- 2022-09-27T16:11:44.2342174Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:11:44.2342542Z 2022-09-27T16:11:44.2342894Z ---------------------------------------------------------------------- 2022-09-27T16:11:44.2343203Z Ran 1 test in 0.000s 2022-09-27T16:11:44.2343328Z 2022-09-27T16:11:44.2343407Z OK (skipped=1) 2022-09-27T16:11:44.2343524Z 2022-09-27T16:11:44.2343598Z Generating XML reports... 2022-09-27T16:11:44.2367252Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161144.xml 2022-09-27T16:11:45.5033868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:45.5034267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:45.5036753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:45.5037109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:45.6412396Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:45.6421857Z 2022-09-27T16:11:45.6421973Z Running tests... 2022-09-27T16:11:45.6422436Z ---------------------------------------------------------------------- 2022-09-27T16:11:45.6426721Z test_nncl_rank_membership (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:11:45.6427120Z 2022-09-27T16:11:45.6427518Z ---------------------------------------------------------------------- 2022-09-27T16:11:45.6427841Z Ran 1 test in 0.000s 2022-09-27T16:11:45.6427959Z 2022-09-27T16:11:45.6428020Z OK (skipped=1) 2022-09-27T16:11:45.6428130Z 2022-09-27T16:11:45.6428216Z Generating XML reports... 2022-09-27T16:11:45.6453187Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161145.xml 2022-09-27T16:11:46.9231512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:46.9231889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:46.9234314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:46.9234867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:47.0614836Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:47.0624562Z 2022-09-27T16:11:47.0624701Z Running tests... 2022-09-27T16:11:47.0625129Z ---------------------------------------------------------------------- 2022-09-27T16:11:47.0635027Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:11:47.0635406Z 2022-09-27T16:11:47.0635637Z ---------------------------------------------------------------------- 2022-09-27T16:11:47.0635874Z Ran 1 test in 0.001s 2022-09-27T16:11:47.0635992Z 2022-09-27T16:11:47.0636069Z OK (skipped=1) 2022-09-27T16:11:47.0636183Z 2022-09-27T16:11:47.0636275Z Generating XML reports... 2022-09-27T16:11:47.0662212Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161147.xml 2022-09-27T16:11:48.3683850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:48.3684500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:48.3686890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:48.3687237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:48.5047178Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:48.5056955Z 2022-09-27T16:11:48.5057052Z Running tests... 2022-09-27T16:11:48.5057491Z ---------------------------------------------------------------------- 2022-09-27T16:11:48.5061448Z test_sequence_num_incremented_nccl_default (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:11:48.5061822Z 2022-09-27T16:11:48.5062226Z ---------------------------------------------------------------------- 2022-09-27T16:11:48.5062612Z Ran 1 test in 0.000s 2022-09-27T16:11:48.5062729Z 2022-09-27T16:11:48.5062790Z OK (skipped=1) 2022-09-27T16:11:48.5062902Z 2022-09-27T16:11:48.5062988Z Generating XML reports... 2022-09-27T16:11:48.5086724Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161148.xml 2022-09-27T16:11:49.7751121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:49.7751494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:49.7754075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:49.7754437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:49.9117942Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:49.9127376Z 2022-09-27T16:11:49.9127480Z Running tests... 2022-09-27T16:11:49.9128072Z ---------------------------------------------------------------------- 2022-09-27T16:11:49.9133106Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:11:49.9133457Z 2022-09-27T16:11:49.9133726Z ---------------------------------------------------------------------- 2022-09-27T16:11:49.9133974Z Ran 1 test in 0.000s 2022-09-27T16:11:49.9134089Z 2022-09-27T16:11:49.9134163Z OK (skipped=1) 2022-09-27T16:11:49.9134272Z 2022-09-27T16:11:49.9134345Z Generating XML reports... 2022-09-27T16:11:49.9158056Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161149.xml 2022-09-27T16:11:51.1889207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:51.1889596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:51.1892384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:51.1892741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:51.3261956Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:51.3270885Z 2022-09-27T16:11:51.3270981Z Running tests... 2022-09-27T16:11:51.3271437Z ---------------------------------------------------------------------- 2022-09-27T16:11:51.3276734Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:11:51.3277188Z 2022-09-27T16:11:51.3277617Z ---------------------------------------------------------------------- 2022-09-27T16:11:51.3278061Z Ran 1 test in 0.001s 2022-09-27T16:11:51.3278270Z 2022-09-27T16:11:51.3278629Z OK (skipped=1) 2022-09-27T16:11:51.3278743Z 2022-09-27T16:11:51.3278816Z Generating XML reports... 2022-09-27T16:11:51.3301516Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161151.xml 2022-09-27T16:11:52.6165656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:52.6166044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:52.6168284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:52.6168800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:52.7526956Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:52.7535685Z 2022-09-27T16:11:52.7535810Z Running tests... 2022-09-27T16:11:52.7536375Z ---------------------------------------------------------------------- 2022-09-27T16:11:52.7540916Z test_sequence_num_set_nccl_new_group (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:11:52.7541288Z 2022-09-27T16:11:52.7541650Z ---------------------------------------------------------------------- 2022-09-27T16:11:52.7542085Z Ran 1 test in 0.000s 2022-09-27T16:11:52.7542322Z 2022-09-27T16:11:52.7542452Z OK (skipped=1) 2022-09-27T16:11:52.7542646Z 2022-09-27T16:11:52.7542806Z Generating XML reports... 2022-09-27T16:11:52.7566035Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161152.xml 2022-09-27T16:11:54.0533359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:54.0533728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:54.0536218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:54.0536594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:54.1899622Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:54.1910441Z 2022-09-27T16:11:54.1910901Z Running tests... 2022-09-27T16:11:54.1911269Z ---------------------------------------------------------------------- 2022-09-27T16:11:54.1915499Z test_tensor_dtype_complex (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:11:54.1915798Z 2022-09-27T16:11:54.1916169Z ---------------------------------------------------------------------- 2022-09-27T16:11:54.1916629Z Ran 1 test in 0.000s 2022-09-27T16:11:54.1916840Z 2022-09-27T16:11:54.1916979Z OK (skipped=1) 2022-09-27T16:11:54.1917099Z 2022-09-27T16:11:54.1917186Z Generating XML reports... 2022-09-27T16:11:54.1940923Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161154.xml 2022-09-27T16:11:55.4942558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:55.4942949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:55.4945493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:55.4946051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:55.6308862Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:55.6318198Z 2022-09-27T16:11:55.6318347Z Running tests... 2022-09-27T16:11:55.6318934Z ---------------------------------------------------------------------- 2022-09-27T16:11:55.6322905Z test_tensor_dtype_mismatch (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:11:55.6323177Z 2022-09-27T16:11:55.6323519Z ---------------------------------------------------------------------- 2022-09-27T16:11:55.6324046Z Ran 1 test in 0.000s 2022-09-27T16:11:55.6324156Z 2022-09-27T16:11:55.6324230Z OK (skipped=1) 2022-09-27T16:11:55.6324407Z 2022-09-27T16:11:55.6324493Z Generating XML reports... 2022-09-27T16:11:55.6347807Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161155.xml 2022-09-27T16:11:56.9487959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:56.9488333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:56.9491004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:56.9491354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:57.0868309Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:11:57.0877793Z 2022-09-27T16:11:57.0877970Z Running tests... 2022-09-27T16:11:57.0879109Z ---------------------------------------------------------------------- 2022-09-27T16:11:57.3340249Z test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8361 2022-09-27T16:11:57.3367112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8362 2022-09-27T16:11:58.4248702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:58.4249081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:58.4251922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:58.4252270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:58.4640608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:11:58.4644262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:11:58.4644726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:11:58.4645074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:11:58.5583114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:11:58.5982826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:11:58.8402140Z skip: Need at least 2 CUDA devices (1.752s) 2022-09-27T16:11:58.8402547Z 2022-09-27T16:11:58.8403065Z ---------------------------------------------------------------------- 2022-09-27T16:11:58.8403355Z Ran 1 test in 1.752s 2022-09-27T16:11:58.8403473Z 2022-09-27T16:11:58.8403549Z OK (skipped=1) 2022-09-27T16:11:58.8403656Z 2022-09-27T16:11:58.8403744Z Generating XML reports... 2022-09-27T16:11:58.8437317Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161157.xml 2022-09-27T16:12:00.2696917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:00.2697314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:00.2699718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:00.2700300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:00.4070120Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:00.4080268Z 2022-09-27T16:12:00.4080434Z Running tests... 2022-09-27T16:12:00.4080847Z ---------------------------------------------------------------------- 2022-09-27T16:12:00.6505149Z test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8394 2022-09-27T16:12:00.6530469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8395 2022-09-27T16:12:01.7913054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:01.7913482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:01.7916787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:01.7917356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:01.7921258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:01.7921647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:01.7925474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:01.7925879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:01.9300056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:01.9300904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:02.2568511Z skip: Need at least 2 CUDA devices (1.849s) 2022-09-27T16:12:02.2568764Z 2022-09-27T16:12:02.2569066Z ---------------------------------------------------------------------- 2022-09-27T16:12:02.2569320Z Ran 1 test in 1.849s 2022-09-27T16:12:02.2569479Z 2022-09-27T16:12:02.2569541Z OK (skipped=1) 2022-09-27T16:12:02.2569652Z 2022-09-27T16:12:02.2569738Z Generating XML reports... 2022-09-27T16:12:02.2603869Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161200.xml 2022-09-27T16:12:03.6944337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:03.6944741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:03.6947390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:03.6947757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:03.8299949Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:03.8309264Z 2022-09-27T16:12:03.8309403Z Running tests... 2022-09-27T16:12:03.8309999Z ---------------------------------------------------------------------- 2022-09-27T16:12:04.0690958Z test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8427 2022-09-27T16:12:04.0717953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8428 2022-09-27T16:12:05.1606782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:05.1607227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:05.1610562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:05.1611084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:05.1907535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:05.1907946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:05.1911436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:05.1911802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:05.2964601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:05.3253866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:05.5756053Z skip: Need at least 2 CUDA devices (1.744s) 2022-09-27T16:12:05.5756537Z 2022-09-27T16:12:05.5757138Z ---------------------------------------------------------------------- 2022-09-27T16:12:05.5757466Z Ran 1 test in 1.745s 2022-09-27T16:12:05.5757594Z 2022-09-27T16:12:05.5757674Z OK (skipped=1) 2022-09-27T16:12:05.5757774Z 2022-09-27T16:12:05.5757860Z Generating XML reports... 2022-09-27T16:12:05.5791720Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161203.xml 2022-09-27T16:12:07.0091177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:07.0091823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:07.0095030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:07.0095610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:07.1506256Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:07.1516389Z 2022-09-27T16:12:07.1516802Z Running tests... 2022-09-27T16:12:07.1517450Z ---------------------------------------------------------------------- 2022-09-27T16:12:07.3984682Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8460 2022-09-27T16:12:07.4011020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8461 2022-09-27T16:12:08.4876352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:08.4877004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:08.4877826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:08.4878405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:08.4879335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:08.4879926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:08.4880720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:08.4881308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:08.6226747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:08.6228895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:08.9046559Z skip: Need at least 2 CUDA devices (1.753s) 2022-09-27T16:12:08.9046830Z 2022-09-27T16:12:08.9047387Z ---------------------------------------------------------------------- 2022-09-27T16:12:08.9047655Z Ran 1 test in 1.753s 2022-09-27T16:12:08.9047774Z 2022-09-27T16:12:08.9047849Z OK (skipped=1) 2022-09-27T16:12:08.9047968Z 2022-09-27T16:12:08.9048054Z Generating XML reports... 2022-09-27T16:12:08.9082702Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161207.xml 2022-09-27T16:12:10.3536419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:10.3536794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:10.3539419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:10.3539779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:10.4914865Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:10.4924242Z 2022-09-27T16:12:10.4924687Z Running tests... 2022-09-27T16:12:10.4925019Z ---------------------------------------------------------------------- 2022-09-27T16:12:10.7384239Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8493 2022-09-27T16:12:10.7410604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8494 2022-09-27T16:12:11.8214031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:11.8214498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:11.8217476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:11.8217843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:11.8541566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:11.8541926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:11.8545463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:11.8545824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:11.9575471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:11.9895519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:12.2449023Z skip: Need at least 2 CUDA devices (1.752s) 2022-09-27T16:12:12.2449479Z 2022-09-27T16:12:12.2450279Z ---------------------------------------------------------------------- 2022-09-27T16:12:12.2450725Z Ran 1 test in 1.752s 2022-09-27T16:12:12.2450907Z 2022-09-27T16:12:12.2451043Z OK (skipped=1) 2022-09-27T16:12:12.2451225Z 2022-09-27T16:12:12.2451355Z Generating XML reports... 2022-09-27T16:12:12.2486235Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161210.xml 2022-09-27T16:12:13.6751553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:13.6752229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:13.6755152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:13.6755735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:13.8116352Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:13.8126402Z 2022-09-27T16:12:13.8126715Z Running tests... 2022-09-27T16:12:13.8127312Z ---------------------------------------------------------------------- 2022-09-27T16:12:14.0625019Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8526 2022-09-27T16:12:14.0651254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8527 2022-09-27T16:12:15.1413384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:15.1413763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:15.1416293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:15.1416752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:15.1417206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:15.1417598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:15.1421334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:15.1421737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:15.2760143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:15.2760543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:15.5686376Z skip: Need at least 2 CUDA devices (1.756s) 2022-09-27T16:12:15.5686678Z 2022-09-27T16:12:15.5687169Z ---------------------------------------------------------------------- 2022-09-27T16:12:15.5687517Z Ran 1 test in 1.756s 2022-09-27T16:12:15.5687678Z 2022-09-27T16:12:15.5687753Z OK (skipped=1) 2022-09-27T16:12:15.5687891Z 2022-09-27T16:12:15.5688330Z Generating XML reports... 2022-09-27T16:12:15.5721681Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161213.xml 2022-09-27T16:12:17.0041060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:17.0041455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:17.0044512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:17.0044898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:17.1390986Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:17.1401549Z 2022-09-27T16:12:17.1401670Z Running tests... 2022-09-27T16:12:17.1402269Z ---------------------------------------------------------------------- 2022-09-27T16:12:17.3877772Z test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8559 2022-09-27T16:12:17.3903482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8560 2022-09-27T16:12:18.4719783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:18.4720491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:18.4722169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:18.4722673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:18.4939339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:18.4939739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:18.4942655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:18.4943136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:18.6064456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:18.6274925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:18.8938582Z skip: Need at least 2 CUDA devices (1.753s) 2022-09-27T16:12:18.8938905Z 2022-09-27T16:12:18.8939261Z ---------------------------------------------------------------------- 2022-09-27T16:12:18.8939510Z Ran 1 test in 1.754s 2022-09-27T16:12:18.8939612Z 2022-09-27T16:12:18.8939686Z OK (skipped=1) 2022-09-27T16:12:18.8939797Z 2022-09-27T16:12:18.8939884Z Generating XML reports... 2022-09-27T16:12:18.8976397Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161217.xml 2022-09-27T16:12:20.3086267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:20.3086650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:20.3089468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:20.3089820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:20.4450465Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:20.4458926Z 2022-09-27T16:12:20.4459073Z Running tests... 2022-09-27T16:12:20.4459674Z ---------------------------------------------------------------------- 2022-09-27T16:12:20.4463403Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:12:20.4464084Z 2022-09-27T16:12:20.4464397Z ---------------------------------------------------------------------- 2022-09-27T16:12:20.4464853Z Ran 1 test in 0.000s 2022-09-27T16:12:20.4465061Z 2022-09-27T16:12:20.4465180Z OK (skipped=1) 2022-09-27T16:12:20.4465291Z 2022-09-27T16:12:20.4465386Z Generating XML reports... 2022-09-27T16:12:20.4490438Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161220.xml 2022-09-27T16:12:21.7358158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:21.7358626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:21.7360332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:21.7360855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:21.8728964Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:21.8738190Z 2022-09-27T16:12:21.8738621Z Running tests... 2022-09-27T16:12:21.8739182Z ---------------------------------------------------------------------- 2022-09-27T16:12:21.8743285Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:12:21.8743787Z 2022-09-27T16:12:21.8744141Z ---------------------------------------------------------------------- 2022-09-27T16:12:21.8744383Z Ran 1 test in 0.000s 2022-09-27T16:12:21.8744500Z 2022-09-27T16:12:21.8744574Z OK (skipped=1) 2022-09-27T16:12:21.8744684Z 2022-09-27T16:12:21.8744770Z Generating XML reports... 2022-09-27T16:12:21.8768434Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161221.xml 2022-09-27T16:12:23.1872038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:23.1872415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:23.1874993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:23.1875377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:23.3246307Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:23.3255487Z 2022-09-27T16:12:23.3255593Z Running tests... 2022-09-27T16:12:23.3256017Z ---------------------------------------------------------------------- 2022-09-27T16:12:23.3260056Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:12:23.3260466Z 2022-09-27T16:12:23.3260909Z ---------------------------------------------------------------------- 2022-09-27T16:12:23.3261275Z Ran 1 test in 0.000s 2022-09-27T16:12:23.3261392Z 2022-09-27T16:12:23.3261451Z OK (skipped=1) 2022-09-27T16:12:23.3261561Z 2022-09-27T16:12:23.3261652Z Generating XML reports... 2022-09-27T16:12:23.3285503Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161223.xml 2022-09-27T16:12:24.6225762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:24.6226317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:24.6238073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:24.6249433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:24.7598727Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:24.7608361Z 2022-09-27T16:12:24.7608802Z Running tests... 2022-09-27T16:12:24.7609194Z ---------------------------------------------------------------------- 2022-09-27T16:12:24.7613176Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:12:24.7613491Z 2022-09-27T16:12:24.7613792Z ---------------------------------------------------------------------- 2022-09-27T16:12:24.7614234Z Ran 1 test in 0.001s 2022-09-27T16:12:24.7614439Z 2022-09-27T16:12:24.7614555Z OK (skipped=1) 2022-09-27T16:12:24.7614746Z 2022-09-27T16:12:24.7614920Z Generating XML reports... 2022-09-27T16:12:24.7639701Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161224.xml 2022-09-27T16:12:26.0525235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:26.0525898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:26.0528674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:26.0529247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:26.1878691Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:26.1887917Z 2022-09-27T16:12:26.1888209Z Running tests... 2022-09-27T16:12:26.1888850Z ---------------------------------------------------------------------- 2022-09-27T16:12:26.1894288Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:12:26.1894743Z 2022-09-27T16:12:26.1895124Z ---------------------------------------------------------------------- 2022-09-27T16:12:26.1895530Z Ran 1 test in 0.001s 2022-09-27T16:12:26.1895717Z 2022-09-27T16:12:26.1895823Z OK (skipped=1) 2022-09-27T16:12:26.1896011Z 2022-09-27T16:12:26.1896161Z Generating XML reports... 2022-09-27T16:12:26.1920734Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161226.xml 2022-09-27T16:12:27.4710980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:27.4711362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:27.4713874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:27.4714366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:27.6081328Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:27.6089542Z 2022-09-27T16:12:27.6089675Z Running tests... 2022-09-27T16:12:27.6090228Z ---------------------------------------------------------------------- 2022-09-27T16:12:27.6095193Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:12:27.6095600Z 2022-09-27T16:12:27.6095962Z ---------------------------------------------------------------------- 2022-09-27T16:12:27.6096402Z Ran 1 test in 0.001s 2022-09-27T16:12:27.6096516Z 2022-09-27T16:12:27.6096589Z OK (skipped=1) 2022-09-27T16:12:27.6096698Z 2022-09-27T16:12:27.6096783Z Generating XML reports... 2022-09-27T16:12:27.6121033Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161227.xml 2022-09-27T16:12:28.9176139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:28.9176522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:28.9178637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:28.9179007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:29.0538381Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:29.0548180Z 2022-09-27T16:12:29.0548334Z Running tests... 2022-09-27T16:12:29.0548767Z ---------------------------------------------------------------------- 2022-09-27T16:12:29.0553040Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:12:29.0553512Z 2022-09-27T16:12:29.0553957Z ---------------------------------------------------------------------- 2022-09-27T16:12:29.0554252Z Ran 1 test in 0.000s 2022-09-27T16:12:29.0554357Z 2022-09-27T16:12:29.0554429Z OK (skipped=1) 2022-09-27T16:12:29.0554684Z 2022-09-27T16:12:29.0554772Z Generating XML reports... 2022-09-27T16:12:29.0578175Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161229.xml 2022-09-27T16:12:30.3548887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:30.3549267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:30.3551016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:30.3551378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:30.4926148Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:30.4935518Z 2022-09-27T16:12:30.4935622Z Running tests... 2022-09-27T16:12:30.4936027Z ---------------------------------------------------------------------- 2022-09-27T16:12:30.4940745Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:12:30.4941098Z 2022-09-27T16:12:30.4941330Z ---------------------------------------------------------------------- 2022-09-27T16:12:30.4941619Z Ran 1 test in 0.000s 2022-09-27T16:12:30.4941734Z 2022-09-27T16:12:30.4941808Z OK (skipped=1) 2022-09-27T16:12:30.4941905Z 2022-09-27T16:12:30.4941990Z Generating XML reports... 2022-09-27T16:12:30.4967062Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161230.xml 2022-09-27T16:12:31.7757705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:31.7758082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:31.7760420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:31.7760769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:31.9130566Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:31.9140177Z 2022-09-27T16:12:31.9140577Z Running tests... 2022-09-27T16:12:31.9141034Z ---------------------------------------------------------------------- 2022-09-27T16:12:31.9147196Z test_channels_last_contig (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:12:31.9147573Z 2022-09-27T16:12:31.9148168Z ---------------------------------------------------------------------- 2022-09-27T16:12:31.9148424Z Ran 1 test in 0.001s 2022-09-27T16:12:31.9148548Z 2022-09-27T16:12:31.9148624Z OK (skipped=1) 2022-09-27T16:12:31.9148810Z 2022-09-27T16:12:31.9148884Z Generating XML reports... 2022-09-27T16:12:31.9172725Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161231.xml 2022-09-27T16:12:33.2099899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:33.2100541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:33.2104347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:33.2104972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:33.3476663Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:33.3487000Z 2022-09-27T16:12:33.3487330Z Running tests... 2022-09-27T16:12:33.3487890Z ---------------------------------------------------------------------- 2022-09-27T16:12:33.3495025Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-09-27T16:12:33.5924858Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8682 2022-09-27T16:12:33.5951144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8683 2022-09-27T16:12:34.6694646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:34.6695015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:34.6697755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:34.6698126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:34.6732938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:34.6733293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:34.6737316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:34.6737671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:34.8021934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:34.8061323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:35.0984983Z skip: Need at least 2 CUDA devices (1.749s) 2022-09-27T16:12:35.0985164Z 2022-09-27T16:12:35.0985623Z ---------------------------------------------------------------------- 2022-09-27T16:12:35.0986085Z Ran 1 test in 1.750s 2022-09-27T16:12:35.0986251Z 2022-09-27T16:12:35.0986339Z OK (skipped=1) 2022-09-27T16:12:35.0986448Z 2022-09-27T16:12:35.0986520Z Generating XML reports... 2022-09-27T16:12:35.1020413Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161233.xml 2022-09-27T16:12:36.4837955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:36.4838375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:36.4840153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:36.4840514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:36.6201092Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:36.6210790Z 2022-09-27T16:12:36.6210904Z Running tests... 2022-09-27T16:12:36.6211493Z ---------------------------------------------------------------------- 2022-09-27T16:12:36.6217754Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-09-27T16:12:36.8613336Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8715 2022-09-27T16:12:36.8641130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8716 2022-09-27T16:12:37.9593357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:37.9594003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:37.9597578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:37.9598137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:37.9742775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:37.9743392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:37.9746824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:37.9747439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:38.0929860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:38.1063262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:38.3678180Z skip: Need at least 2 CUDA devices (1.746s) 2022-09-27T16:12:38.3678440Z 2022-09-27T16:12:38.3678867Z ---------------------------------------------------------------------- 2022-09-27T16:12:38.3679268Z Ran 1 test in 1.747s 2022-09-27T16:12:38.3679444Z 2022-09-27T16:12:38.3679557Z OK (skipped=1) 2022-09-27T16:12:38.3679732Z 2022-09-27T16:12:38.3679886Z Generating XML reports... 2022-09-27T16:12:38.3714705Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161236.xml 2022-09-27T16:12:39.7608609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:39.7608976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:39.7611503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:39.7612075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:39.8980469Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:39.8989716Z 2022-09-27T16:12:39.8989849Z Running tests... 2022-09-27T16:12:39.8990400Z ---------------------------------------------------------------------- 2022-09-27T16:12:39.8999729Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-09-27T16:12:40.1442097Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8748 2022-09-27T16:12:40.1467754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8749 2022-09-27T16:12:41.2317673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:41.2318055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:41.2321060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:41.2321532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:41.2660557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:41.2661267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:41.2664160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:41.2664721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:41.3669217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:41.3997131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:41.6502258Z skip: Need at least 2 CUDA devices (1.751s) 2022-09-27T16:12:41.6502647Z 2022-09-27T16:12:41.6502995Z ---------------------------------------------------------------------- 2022-09-27T16:12:41.6503331Z Ran 1 test in 1.751s 2022-09-27T16:12:41.6503445Z 2022-09-27T16:12:41.6503521Z OK (skipped=1) 2022-09-27T16:12:41.6503631Z 2022-09-27T16:12:41.6503707Z Generating XML reports... 2022-09-27T16:12:41.6539910Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161239.xml 2022-09-27T16:12:43.0511412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:43.0512069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:43.0515778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:43.0516418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:43.1886322Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:43.1895067Z 2022-09-27T16:12:43.1895195Z Running tests... 2022-09-27T16:12:43.1895795Z ---------------------------------------------------------------------- 2022-09-27T16:12:43.1904451Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-09-27T16:12:43.4341814Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8781 2022-09-27T16:12:43.4368194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8782 2022-09-27T16:12:44.5197490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:44.5197859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:44.5200267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:44.5200627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:44.5290220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:44.5290588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:44.5294237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:44.5294717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:44.6533739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:44.6620952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:44.9403738Z skip: Need at least 2 CUDA devices (1.750s) 2022-09-27T16:12:44.9404277Z 2022-09-27T16:12:44.9404759Z ---------------------------------------------------------------------- 2022-09-27T16:12:44.9405062Z Ran 1 test in 1.751s 2022-09-27T16:12:44.9405178Z 2022-09-27T16:12:44.9405250Z OK (skipped=1) 2022-09-27T16:12:44.9405358Z 2022-09-27T16:12:44.9405442Z Generating XML reports... 2022-09-27T16:12:44.9440177Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161243.xml 2022-09-27T16:12:46.3509411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:46.3509798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:46.3511740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:46.3512087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:46.4874769Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:46.4883714Z 2022-09-27T16:12:46.4883851Z Running tests... 2022-09-27T16:12:46.4884300Z ---------------------------------------------------------------------- 2022-09-27T16:12:46.4891028Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-09-27T16:12:46.7333145Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8814 2022-09-27T16:12:46.7359482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8815 2022-09-27T16:12:47.8542038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:47.8542686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:47.8545675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:47.8546229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:47.8582818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:47.8583403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:47.8586750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:47.8587339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:47.9891503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:47.9939696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:48.2404095Z skip: Need at least 2 CUDA devices (1.751s) 2022-09-27T16:12:48.2404413Z 2022-09-27T16:12:48.2404732Z ---------------------------------------------------------------------- 2022-09-27T16:12:48.2404972Z Ran 1 test in 1.751s 2022-09-27T16:12:48.2405088Z 2022-09-27T16:12:48.2405162Z OK (skipped=1) 2022-09-27T16:12:48.2405271Z 2022-09-27T16:12:48.2405357Z Generating XML reports... 2022-09-27T16:12:48.2434408Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161246.xml 2022-09-27T16:12:49.6639572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:49.6639985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:49.6642390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:49.6642746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:49.8003545Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:49.8013171Z 2022-09-27T16:12:49.8013281Z Running tests... 2022-09-27T16:12:49.8013775Z ---------------------------------------------------------------------- 2022-09-27T16:12:49.8020625Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-09-27T16:12:50.0459233Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8847 2022-09-27T16:12:50.0485080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8848 2022-09-27T16:12:51.1446112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:51.1446938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:51.1448492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:51.1449045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:51.1761764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:51.1762357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:51.1763611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:51.1764210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:51.2779417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:51.3103844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:51.6526126Z skip: Need at least 2 CUDA devices (1.851s) 2022-09-27T16:12:51.6526452Z 2022-09-27T16:12:51.6526948Z ---------------------------------------------------------------------- 2022-09-27T16:12:51.6527395Z Ran 1 test in 1.851s 2022-09-27T16:12:51.6527532Z 2022-09-27T16:12:51.6527611Z OK (skipped=1) 2022-09-27T16:12:51.6527718Z 2022-09-27T16:12:51.6527804Z Generating XML reports... 2022-09-27T16:12:51.6561568Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161249.xml 2022-09-27T16:12:53.0835626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:53.0836010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:53.0838747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:53.0839141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:53.2199147Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:53.2208697Z 2022-09-27T16:12:53.2208794Z Running tests... 2022-09-27T16:12:53.2209345Z ---------------------------------------------------------------------- 2022-09-27T16:12:53.2218736Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-09-27T16:12:53.4670177Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8880 2022-09-27T16:12:53.4696961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8881 2022-09-27T16:12:54.5530064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:54.5530454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:54.5533331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:54.5533968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:54.5534514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:54.5534849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:54.5538161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:54.5538643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:54.6886514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:54.6892380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:54.9732039Z skip: Need at least 2 CUDA devices (1.752s) 2022-09-27T16:12:54.9732350Z 2022-09-27T16:12:54.9732751Z ---------------------------------------------------------------------- 2022-09-27T16:12:54.9733016Z Ran 1 test in 1.752s 2022-09-27T16:12:54.9733130Z 2022-09-27T16:12:54.9733203Z OK (skipped=1) 2022-09-27T16:12:54.9733310Z 2022-09-27T16:12:54.9733393Z Generating XML reports... 2022-09-27T16:12:54.9769407Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161253.xml 2022-09-27T16:12:56.3865640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:56.3866012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:56.3868366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:56.3868890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:56.5231683Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:56.5241364Z 2022-09-27T16:12:56.5241507Z Running tests... 2022-09-27T16:12:56.5242101Z ---------------------------------------------------------------------- 2022-09-27T16:12:56.5251711Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-09-27T16:12:56.7725753Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8913 2022-09-27T16:12:56.7751054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8914 2022-09-27T16:12:57.8823890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:57.8824329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:57.8827090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:57.8827554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:57.8985219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:57.8985609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:57.8988474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:57.8988837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:58.0187542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:12:58.0337045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:12:58.2785945Z skip: Need at least 2 CUDA devices (1.754s) 2022-09-27T16:12:58.2786478Z 2022-09-27T16:12:58.2787096Z ---------------------------------------------------------------------- 2022-09-27T16:12:58.2787534Z Ran 1 test in 1.754s 2022-09-27T16:12:58.2787697Z 2022-09-27T16:12:58.2787816Z OK (skipped=1) 2022-09-27T16:12:58.2787981Z 2022-09-27T16:12:58.2788120Z Generating XML reports... 2022-09-27T16:12:58.2822862Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161256.xml 2022-09-27T16:12:59.7266549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:12:59.7266922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:12:59.7269311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:12:59.7269891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:12:59.8649961Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:12:59.8659373Z 2022-09-27T16:12:59.8659499Z Running tests... 2022-09-27T16:12:59.8659880Z ---------------------------------------------------------------------- 2022-09-27T16:12:59.8666857Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-09-27T16:13:00.1131992Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8946 2022-09-27T16:13:00.1159350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8947 2022-09-27T16:13:01.2275276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:01.2275662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:01.2278262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:01.2278646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:01.2527019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:01.2527367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:01.2530577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:01.2530936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:01.3640397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:13:01.3868170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:13:01.7197133Z skip: Need at least 2 CUDA devices (1.853s) 2022-09-27T16:13:01.7197502Z 2022-09-27T16:13:01.7197812Z ---------------------------------------------------------------------- 2022-09-27T16:13:01.7198054Z Ran 1 test in 1.854s 2022-09-27T16:13:01.7198167Z 2022-09-27T16:13:01.7198240Z OK (skipped=1) 2022-09-27T16:13:01.7198348Z 2022-09-27T16:13:01.7198434Z Generating XML reports... 2022-09-27T16:13:01.7233076Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161259.xml 2022-09-27T16:13:03.1969659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:03.1970049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:03.1972657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:03.1973049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:03.3357944Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:03.3367472Z 2022-09-27T16:13:03.3367607Z Running tests... 2022-09-27T16:13:03.3368201Z ---------------------------------------------------------------------- 2022-09-27T16:13:03.3377541Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-09-27T16:13:03.5841304Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8979 2022-09-27T16:13:03.5868525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8980 2022-09-27T16:13:04.6906343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:04.6906718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:04.6910081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:04.6910451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:04.7261314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:04.7261662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:04.7264746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:04.7265133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:04.8267869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:13:04.8609219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:13:05.1907030Z skip: Need at least 2 CUDA devices (1.854s) 2022-09-27T16:13:05.1907261Z 2022-09-27T16:13:05.1907602Z ---------------------------------------------------------------------- 2022-09-27T16:13:05.1907898Z Ran 1 test in 1.854s 2022-09-27T16:13:05.1908100Z 2022-09-27T16:13:05.1908212Z OK (skipped=1) 2022-09-27T16:13:05.1908371Z 2022-09-27T16:13:05.1908457Z Generating XML reports... 2022-09-27T16:13:05.1943050Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161303.xml 2022-09-27T16:13:06.6323106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:06.6323488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:06.6325693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:06.6326048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:06.7713269Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:06.7722474Z 2022-09-27T16:13:06.7722637Z Running tests... 2022-09-27T16:13:06.7723854Z ---------------------------------------------------------------------- 2022-09-27T16:13:06.7732090Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-09-27T16:13:07.0225649Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9012 2022-09-27T16:13:07.0252507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9013 2022-09-27T16:13:08.1259121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:08.1259503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:08.1262092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:08.1262484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:08.1598458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:08.1598830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:08.1601903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:08.1602268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:08.2610050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:13:08.2943972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:13:08.6288621Z skip: Need at least 2 CUDA devices (1.856s) 2022-09-27T16:13:08.6288911Z 2022-09-27T16:13:08.6289397Z ---------------------------------------------------------------------- 2022-09-27T16:13:08.6289818Z Ran 1 test in 1.857s 2022-09-27T16:13:08.6290331Z 2022-09-27T16:13:08.6290460Z OK (skipped=1) 2022-09-27T16:13:08.6290651Z 2022-09-27T16:13:08.6290782Z Generating XML reports... 2022-09-27T16:13:08.6324288Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161306.xml 2022-09-27T16:13:10.0756160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:10.0756534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:10.0759100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:10.0759553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:10.2140530Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:10.2150070Z 2022-09-27T16:13:10.2150170Z Running tests... 2022-09-27T16:13:10.2150617Z ---------------------------------------------------------------------- 2022-09-27T16:13:10.2161651Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-09-27T16:13:10.4665049Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9045 2022-09-27T16:13:10.4692305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9046 2022-09-27T16:13:11.5737971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:11.5738602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:11.5741565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:11.5742125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:11.5849473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:11.5850088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:11.5853034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:11.5853625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:11.7088357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:13:11.7187467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:13:11.9728327Z skip: Need at least 2 CUDA devices (1.757s) 2022-09-27T16:13:11.9728647Z 2022-09-27T16:13:11.9729134Z ---------------------------------------------------------------------- 2022-09-27T16:13:11.9729395Z Ran 1 test in 1.758s 2022-09-27T16:13:11.9729498Z 2022-09-27T16:13:11.9729569Z OK (skipped=1) 2022-09-27T16:13:11.9729678Z 2022-09-27T16:13:11.9729777Z Generating XML reports... 2022-09-27T16:13:11.9764005Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161310.xml 2022-09-27T16:13:13.4347543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:13.4348117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:13.4349845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:13.4350418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:13.5747569Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:13.5757509Z 2022-09-27T16:13:13.5757796Z Running tests... 2022-09-27T16:13:13.5769936Z ---------------------------------------------------------------------- 2022-09-27T16:13:13.5770831Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-09-27T16:13:13.8351833Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9078 2022-09-27T16:13:13.8385529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9079 2022-09-27T16:13:14.9460951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:14.9461343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:14.9463802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:14.9464259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:14.9648040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:14.9648587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:14.9651326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:14.9651682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:15.0829365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:13:15.1007544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:13:15.3420657Z skip: Need at least 2 CUDA devices (1.766s) 2022-09-27T16:13:15.3420828Z 2022-09-27T16:13:15.3421139Z ---------------------------------------------------------------------- 2022-09-27T16:13:15.3421432Z Ran 1 test in 1.766s 2022-09-27T16:13:15.3421563Z 2022-09-27T16:13:15.3421633Z OK (skipped=1) 2022-09-27T16:13:15.3421739Z 2022-09-27T16:13:15.3421810Z Generating XML reports... 2022-09-27T16:13:15.3455846Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161313.xml 2022-09-27T16:13:16.8002950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:16.8003353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:16.8005822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:16.8006182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:16.9501358Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:16.9510343Z 2022-09-27T16:13:16.9510856Z Running tests... 2022-09-27T16:13:16.9511268Z ---------------------------------------------------------------------- 2022-09-27T16:13:16.9514821Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:16.9515324Z 2022-09-27T16:13:16.9515754Z ---------------------------------------------------------------------- 2022-09-27T16:13:16.9516031Z Ran 1 test in 0.000s 2022-09-27T16:13:16.9516140Z 2022-09-27T16:13:16.9516215Z OK (skipped=1) 2022-09-27T16:13:16.9516324Z 2022-09-27T16:13:16.9516411Z Generating XML reports... 2022-09-27T16:13:16.9542173Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161316.xml 2022-09-27T16:13:18.2596755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:18.2597156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:18.2599552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:18.2599976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:18.3977694Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:18.3986936Z 2022-09-27T16:13:18.3987184Z Running tests... 2022-09-27T16:13:18.3987638Z ---------------------------------------------------------------------- 2022-09-27T16:13:18.3991639Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:18.3992091Z 2022-09-27T16:13:18.3992554Z ---------------------------------------------------------------------- 2022-09-27T16:13:18.3992943Z Ran 1 test in 0.000s 2022-09-27T16:13:18.3993065Z 2022-09-27T16:13:18.3993138Z OK (skipped=1) 2022-09-27T16:13:18.3993248Z 2022-09-27T16:13:18.3993333Z Generating XML reports... 2022-09-27T16:13:18.4017291Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161318.xml 2022-09-27T16:13:19.7250621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:19.7250984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:19.7253682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:19.7254027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:19.8625072Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:19.8635212Z 2022-09-27T16:13:19.8635326Z Running tests... 2022-09-27T16:13:19.8635792Z ---------------------------------------------------------------------- 2022-09-27T16:13:19.8640535Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:19.8640997Z 2022-09-27T16:13:19.8641463Z ---------------------------------------------------------------------- 2022-09-27T16:13:19.8641855Z Ran 1 test in 0.001s 2022-09-27T16:13:19.8641970Z 2022-09-27T16:13:19.8642042Z OK (skipped=1) 2022-09-27T16:13:19.8642159Z 2022-09-27T16:13:19.8642232Z Generating XML reports... 2022-09-27T16:13:19.8667165Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161319.xml 2022-09-27T16:13:21.1658435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:21.1658810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:21.1661222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:21.1661577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:21.3039491Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:21.3048968Z 2022-09-27T16:13:21.3049098Z Running tests... 2022-09-27T16:13:21.3049661Z ---------------------------------------------------------------------- 2022-09-27T16:13:21.3060649Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-09-27T16:13:21.3061679Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:13:21.3062099Z 2022-09-27T16:13:21.3062544Z ---------------------------------------------------------------------- 2022-09-27T16:13:21.3063003Z Ran 1 test in 0.001s 2022-09-27T16:13:21.3063194Z 2022-09-27T16:13:21.3063287Z OK (skipped=1) 2022-09-27T16:13:21.3063385Z 2022-09-27T16:13:21.3063472Z Generating XML reports... 2022-09-27T16:13:21.3089418Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161321.xml 2022-09-27T16:13:22.5940526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:22.5940914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:22.5943159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:22.5943713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:22.7309922Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:22.7320217Z 2022-09-27T16:13:22.7320359Z Running tests... 2022-09-27T16:13:22.7321235Z ---------------------------------------------------------------------- 2022-09-27T16:13:22.7328440Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-09-27T16:13:22.7329033Z This unit test verifies whether the Future object is passed properly using nccl backend. ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:13:22.7329500Z 2022-09-27T16:13:22.7329973Z ---------------------------------------------------------------------- 2022-09-27T16:13:22.7330401Z Ran 1 test in 0.001s 2022-09-27T16:13:22.7330620Z 2022-09-27T16:13:22.7330741Z OK (skipped=1) 2022-09-27T16:13:22.7330843Z 2022-09-27T16:13:22.7330929Z Generating XML reports... 2022-09-27T16:13:22.7354281Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161322.xml 2022-09-27T16:13:24.0379881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:24.0380306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:24.0382384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:24.0382801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:24.1774761Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:24.1783616Z 2022-09-27T16:13:24.1783787Z Running tests... 2022-09-27T16:13:24.1784406Z ---------------------------------------------------------------------- 2022-09-27T16:13:24.1796276Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:13:24.1796647Z 2022-09-27T16:13:24.1796972Z ---------------------------------------------------------------------- 2022-09-27T16:13:24.1797324Z Ran 1 test in 0.001s 2022-09-27T16:13:24.1797472Z 2022-09-27T16:13:24.1797545Z OK (skipped=1) 2022-09-27T16:13:24.1797654Z 2022-09-27T16:13:24.1799642Z Generating XML reports... 2022-09-27T16:13:24.1822038Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161324.xml 2022-09-27T16:13:25.4981486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:25.4981875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:25.4983861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:25.4984297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:25.6332420Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:25.6341309Z 2022-09-27T16:13:25.6341455Z Running tests... 2022-09-27T16:13:25.6341902Z ---------------------------------------------------------------------- 2022-09-27T16:13:25.6356688Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:13:25.6357147Z 2022-09-27T16:13:25.6357582Z ---------------------------------------------------------------------- 2022-09-27T16:13:25.6358019Z Ran 1 test in 0.001s 2022-09-27T16:13:25.6358221Z 2022-09-27T16:13:25.6358621Z OK (skipped=1) 2022-09-27T16:13:25.6358800Z 2022-09-27T16:13:25.6358902Z Generating XML reports... 2022-09-27T16:13:25.6381370Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161325.xml 2022-09-27T16:13:26.9340811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:26.9341198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:26.9343671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:26.9344029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:27.0713254Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:27.0722914Z 2022-09-27T16:13:27.0723016Z Running tests... 2022-09-27T16:13:27.0723428Z ---------------------------------------------------------------------- 2022-09-27T16:13:27.0729498Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:13:27.0729782Z 2022-09-27T16:13:27.0730137Z ---------------------------------------------------------------------- 2022-09-27T16:13:27.0730586Z Ran 1 test in 0.001s 2022-09-27T16:13:27.0730757Z 2022-09-27T16:13:27.0730818Z OK (skipped=1) 2022-09-27T16:13:27.0730926Z 2022-09-27T16:13:27.0731014Z Generating XML reports... 2022-09-27T16:13:27.0754689Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161327.xml 2022-09-27T16:13:28.3811466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:28.3811841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:28.3814114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:28.3814461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:28.5173673Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:28.5182298Z 2022-09-27T16:13:28.5182383Z Running tests... 2022-09-27T16:13:28.5183571Z ---------------------------------------------------------------------- 2022-09-27T16:13:28.5186928Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:28.5187201Z 2022-09-27T16:13:28.5187557Z ---------------------------------------------------------------------- 2022-09-27T16:13:28.5188010Z Ran 1 test in 0.000s 2022-09-27T16:13:28.5188216Z 2022-09-27T16:13:28.5188322Z OK (skipped=1) 2022-09-27T16:13:28.5188432Z 2022-09-27T16:13:28.5188504Z Generating XML reports... 2022-09-27T16:13:28.5213564Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161328.xml 2022-09-27T16:13:29.8175449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:29.8175933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:29.8178183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:29.8178551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:29.9535179Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:29.9544476Z 2022-09-27T16:13:29.9544638Z Running tests... 2022-09-27T16:13:29.9544977Z ---------------------------------------------------------------------- 2022-09-27T16:13:29.9549198Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:29.9549839Z 2022-09-27T16:13:29.9550282Z ---------------------------------------------------------------------- 2022-09-27T16:13:29.9550754Z Ran 1 test in 0.000s 2022-09-27T16:13:29.9550869Z 2022-09-27T16:13:29.9550941Z OK (skipped=1) 2022-09-27T16:13:29.9551048Z 2022-09-27T16:13:29.9551131Z Generating XML reports... 2022-09-27T16:13:29.9574719Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161329.xml 2022-09-27T16:13:31.2463082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:31.2463450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:31.2465841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:31.2466208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:31.3824616Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:31.3834006Z 2022-09-27T16:13:31.3834112Z Running tests... 2022-09-27T16:13:31.3834732Z ---------------------------------------------------------------------- 2022-09-27T16:13:31.3858250Z test_failure_recovery (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:13:31.3858704Z 2022-09-27T16:13:31.3859024Z ---------------------------------------------------------------------- 2022-09-27T16:13:31.3859255Z Ran 1 test in 0.002s 2022-09-27T16:13:31.3859367Z 2022-09-27T16:13:31.3860077Z OK (skipped=1) 2022-09-27T16:13:31.3860232Z 2022-09-27T16:13:31.3860320Z Generating XML reports... 2022-09-27T16:13:31.3882680Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161331.xml 2022-09-27T16:13:32.6603470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:32.6603842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:32.6606357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:32.6606713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:32.7967921Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:32.7977250Z 2022-09-27T16:13:32.7977347Z Running tests... 2022-09-27T16:13:32.7977883Z ---------------------------------------------------------------------- 2022-09-27T16:13:32.7982770Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:32.7983199Z 2022-09-27T16:13:32.7983598Z ---------------------------------------------------------------------- 2022-09-27T16:13:32.7983944Z Ran 1 test in 0.000s 2022-09-27T16:13:32.7984066Z 2022-09-27T16:13:32.7984124Z OK (skipped=1) 2022-09-27T16:13:32.7984231Z 2022-09-27T16:13:32.7984313Z Generating XML reports... 2022-09-27T16:13:32.8007073Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161332.xml 2022-09-27T16:13:34.0758683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:34.0759062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:34.0761384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:34.0761983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:34.2122655Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:34.2131815Z 2022-09-27T16:13:34.2132210Z Running tests... 2022-09-27T16:13:34.2132812Z ---------------------------------------------------------------------- 2022-09-27T16:13:34.2136834Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:34.2137627Z 2022-09-27T16:13:34.2138065Z ---------------------------------------------------------------------- 2022-09-27T16:13:34.2138319Z Ran 1 test in 0.000s 2022-09-27T16:13:34.2138439Z 2022-09-27T16:13:34.2138499Z OK (skipped=1) 2022-09-27T16:13:34.2138609Z 2022-09-27T16:13:34.2138696Z Generating XML reports... 2022-09-27T16:13:34.2162569Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161334.xml 2022-09-27T16:13:35.4867155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:35.4867522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:35.4869899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:35.4870271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:35.6243022Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:35.6252383Z 2022-09-27T16:13:35.6252487Z Running tests... 2022-09-27T16:13:35.6253018Z ---------------------------------------------------------------------- 2022-09-27T16:13:35.6257380Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:35.6258063Z 2022-09-27T16:13:35.6258627Z ---------------------------------------------------------------------- 2022-09-27T16:13:35.6259047Z Ran 1 test in 0.000s 2022-09-27T16:13:35.6259238Z 2022-09-27T16:13:35.6259373Z OK (skipped=1) 2022-09-27T16:13:35.6259559Z 2022-09-27T16:13:35.6259709Z Generating XML reports... 2022-09-27T16:13:35.6284488Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161335.xml 2022-09-27T16:13:36.9015883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:36.9016261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:36.9018901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:36.9019290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:37.0386620Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:37.0396231Z 2022-09-27T16:13:37.0396316Z Running tests... 2022-09-27T16:13:37.0396701Z ---------------------------------------------------------------------- 2022-09-27T16:13:37.0402311Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:37.0402851Z 2022-09-27T16:13:37.0403226Z ---------------------------------------------------------------------- 2022-09-27T16:13:37.0403470Z Ran 1 test in 0.001s 2022-09-27T16:13:37.0403585Z 2022-09-27T16:13:37.0403645Z OK (skipped=1) 2022-09-27T16:13:37.0403752Z 2022-09-27T16:13:37.0403835Z Generating XML reports... 2022-09-27T16:13:37.0426690Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161337.xml 2022-09-27T16:13:38.3363498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:38.3363856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:38.3366187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:38.3366901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:38.4737433Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:38.4746351Z 2022-09-27T16:13:38.4746464Z Running tests... 2022-09-27T16:13:38.4747322Z ---------------------------------------------------------------------- 2022-09-27T16:13:38.4751164Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:38.4751587Z 2022-09-27T16:13:38.4751827Z ---------------------------------------------------------------------- 2022-09-27T16:13:38.4752069Z Ran 1 test in 0.000s 2022-09-27T16:13:38.4752182Z 2022-09-27T16:13:38.4752254Z OK (skipped=1) 2022-09-27T16:13:38.4752348Z 2022-09-27T16:13:38.4752433Z Generating XML reports... 2022-09-27T16:13:38.4776066Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161338.xml 2022-09-27T16:13:39.7604134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:39.7604810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:39.7607650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:39.7608238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:39.8957421Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:39.8967207Z 2022-09-27T16:13:39.8967514Z Running tests... 2022-09-27T16:13:39.8968080Z ---------------------------------------------------------------------- 2022-09-27T16:13:39.8972706Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:39.8973202Z 2022-09-27T16:13:39.8973692Z ---------------------------------------------------------------------- 2022-09-27T16:13:39.8974138Z Ran 1 test in 0.001s 2022-09-27T16:13:39.8974326Z 2022-09-27T16:13:39.8974387Z OK (skipped=1) 2022-09-27T16:13:39.8974492Z 2022-09-27T16:13:39.8974575Z Generating XML reports... 2022-09-27T16:13:39.9000137Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161339.xml 2022-09-27T16:13:41.1986158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:41.1986555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:41.1988834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:41.1989213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:41.3354789Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:41.3363796Z 2022-09-27T16:13:41.3363983Z Running tests... 2022-09-27T16:13:41.3364464Z ---------------------------------------------------------------------- 2022-09-27T16:13:41.3368263Z test_fp16 (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:41.3368615Z 2022-09-27T16:13:41.3368859Z ---------------------------------------------------------------------- 2022-09-27T16:13:41.3369150Z Ran 1 test in 0.000s 2022-09-27T16:13:41.3369324Z 2022-09-27T16:13:41.3369406Z OK (skipped=1) 2022-09-27T16:13:41.3369518Z 2022-09-27T16:13:41.3369590Z Generating XML reports... 2022-09-27T16:13:41.3393182Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161341.xml 2022-09-27T16:13:42.6276474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:42.6277092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:42.6279504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:42.6279980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:42.7640173Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:42.7649651Z 2022-09-27T16:13:42.7649740Z Running tests... 2022-09-27T16:13:42.7650507Z ---------------------------------------------------------------------- 2022-09-27T16:13:42.7654200Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:42.7654764Z 2022-09-27T16:13:42.7655038Z ---------------------------------------------------------------------- 2022-09-27T16:13:42.7655507Z Ran 1 test in 0.000s 2022-09-27T16:13:42.7655719Z 2022-09-27T16:13:42.7655843Z OK (skipped=1) 2022-09-27T16:13:42.7655955Z 2022-09-27T16:13:42.7656034Z Generating XML reports... 2022-09-27T16:13:42.7679929Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161342.xml 2022-09-27T16:13:44.0617614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:44.0617988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:44.0620515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:44.0621033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:44.2000963Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:44.2010833Z 2022-09-27T16:13:44.2011017Z Running tests... 2022-09-27T16:13:44.2011345Z ---------------------------------------------------------------------- 2022-09-27T16:13:44.2015289Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:44.2015758Z 2022-09-27T16:13:44.2016190Z ---------------------------------------------------------------------- 2022-09-27T16:13:44.2016617Z Ran 1 test in 0.000s 2022-09-27T16:13:44.2016828Z 2022-09-27T16:13:44.2016970Z OK (skipped=1) 2022-09-27T16:13:44.2017172Z 2022-09-27T16:13:44.2017311Z Generating XML reports... 2022-09-27T16:13:44.2041880Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161344.xml 2022-09-27T16:13:45.4791873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:45.4792309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:45.4794965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:45.4795325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:45.6156731Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:45.6166841Z 2022-09-27T16:13:45.6166944Z Running tests... 2022-09-27T16:13:45.6167661Z ---------------------------------------------------------------------- 2022-09-27T16:13:45.6171110Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:45.6171479Z 2022-09-27T16:13:45.6171883Z ---------------------------------------------------------------------- 2022-09-27T16:13:45.6172333Z Ran 1 test in 0.000s 2022-09-27T16:13:45.6172520Z 2022-09-27T16:13:45.6172582Z OK (skipped=1) 2022-09-27T16:13:45.6172693Z 2022-09-27T16:13:45.6172781Z Generating XML reports... 2022-09-27T16:13:45.6196463Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161345.xml 2022-09-27T16:13:46.9148559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:46.9149458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:46.9151750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:46.9152309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:47.0534442Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:47.0544445Z 2022-09-27T16:13:47.0544851Z Running tests... 2022-09-27T16:13:47.0545251Z ---------------------------------------------------------------------- 2022-09-27T16:13:47.0550809Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:13:47.0551216Z 2022-09-27T16:13:47.0551853Z ---------------------------------------------------------------------- 2022-09-27T16:13:47.0552288Z Ran 1 test in 0.001s 2022-09-27T16:13:47.0552495Z 2022-09-27T16:13:47.0552596Z OK (skipped=1) 2022-09-27T16:13:47.0552707Z 2022-09-27T16:13:47.0552794Z Generating XML reports... 2022-09-27T16:13:47.0577660Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161347.xml 2022-09-27T16:13:48.3557740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:48.3558127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:48.3560742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:48.3561088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:48.4925985Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:48.4934760Z 2022-09-27T16:13:48.4935033Z Running tests... 2022-09-27T16:13:48.4935666Z ---------------------------------------------------------------------- 2022-09-27T16:13:48.4942275Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:13:48.4942826Z 2022-09-27T16:13:48.4943224Z ---------------------------------------------------------------------- 2022-09-27T16:13:48.4943685Z Ran 1 test in 0.001s 2022-09-27T16:13:48.4943900Z 2022-09-27T16:13:48.4944003Z OK (skipped=1) 2022-09-27T16:13:48.4944111Z 2022-09-27T16:13:48.4944183Z Generating XML reports... 2022-09-27T16:13:48.4968453Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161348.xml 2022-09-27T16:13:49.7979535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:49.7979905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:49.7982489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:49.7982870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:49.9345510Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:49.9356010Z 2022-09-27T16:13:49.9356344Z Running tests... 2022-09-27T16:13:49.9356695Z ---------------------------------------------------------------------- 2022-09-27T16:13:50.1792795Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9341 2022-09-27T16:13:50.1818445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9342 2022-09-27T16:13:51.2741194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:51.2741600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:51.2743859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:51.2744381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:51.2893850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:51.2894268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:51.2897032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:51.2897551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:51.4088222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:13:51.4093264Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-09-27T16:13:51.4094842Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-09-27T16:13:51.4096368Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-09-27T16:13:51.4097859Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-09-27T16:13:51.4099373Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-09-27T16:13:51.4100865Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-09-27T16:13:51.4249491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:13:51.4255190Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-09-27T16:13:51.4256522Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-09-27T16:13:51.4258010Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-09-27T16:13:51.4259704Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-09-27T16:13:51.4261285Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-09-27T16:13:51.4262824Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-09-27T16:13:51.6854133Z ok (1.749s) 2022-09-27T16:13:51.6854419Z 2022-09-27T16:13:51.6854900Z ---------------------------------------------------------------------- 2022-09-27T16:13:51.6855142Z Ran 1 test in 1.750s 2022-09-27T16:13:51.6855262Z 2022-09-27T16:13:51.6855322Z OK 2022-09-27T16:13:51.6855414Z 2022-09-27T16:13:51.6855509Z Generating XML reports... 2022-09-27T16:13:51.6888398Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161349.xml 2022-09-27T16:13:53.0908566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:53.0909226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:53.0912780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:53.0913366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:53.2275439Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:53.2285396Z 2022-09-27T16:13:53.2285859Z Running tests... 2022-09-27T16:13:53.2286408Z ---------------------------------------------------------------------- 2022-09-27T16:13:53.2290405Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:53.2290693Z 2022-09-27T16:13:53.2291044Z ---------------------------------------------------------------------- 2022-09-27T16:13:53.2291308Z Ran 1 test in 0.000s 2022-09-27T16:13:53.2291424Z 2022-09-27T16:13:53.2291499Z OK (skipped=1) 2022-09-27T16:13:53.2291600Z 2022-09-27T16:13:53.2291686Z Generating XML reports... 2022-09-27T16:13:53.2315733Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161353.xml 2022-09-27T16:13:54.5162875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:54.5163522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:54.5165794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:54.5166368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:54.6530520Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:54.6540207Z 2022-09-27T16:13:54.6540741Z Running tests... 2022-09-27T16:13:54.6541222Z ---------------------------------------------------------------------- 2022-09-27T16:13:54.6545914Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:54.6546357Z 2022-09-27T16:13:54.6546782Z ---------------------------------------------------------------------- 2022-09-27T16:13:54.6547095Z Ran 1 test in 0.000s 2022-09-27T16:13:54.6547207Z 2022-09-27T16:13:54.6547279Z OK (skipped=1) 2022-09-27T16:13:54.6547386Z 2022-09-27T16:13:54.6547472Z Generating XML reports... 2022-09-27T16:13:54.6570308Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161354.xml 2022-09-27T16:13:55.9271117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:55.9271554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:55.9273792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:55.9274242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:56.0640601Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:56.0650035Z 2022-09-27T16:13:56.0650248Z Running tests... 2022-09-27T16:13:56.0650594Z ---------------------------------------------------------------------- 2022-09-27T16:13:56.0656687Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:56.0657179Z 2022-09-27T16:13:56.0657574Z ---------------------------------------------------------------------- 2022-09-27T16:13:56.0657978Z Ran 1 test in 0.001s 2022-09-27T16:13:56.0658162Z 2022-09-27T16:13:56.0658285Z OK (skipped=1) 2022-09-27T16:13:56.0658470Z 2022-09-27T16:13:56.0658620Z Generating XML reports... 2022-09-27T16:13:56.0682935Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161356.xml 2022-09-27T16:13:57.3522245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:57.3522844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:57.3525236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:57.3525797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:57.4892322Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:57.4902273Z 2022-09-27T16:13:57.4902726Z Running tests... 2022-09-27T16:13:57.4903203Z ---------------------------------------------------------------------- 2022-09-27T16:13:57.4908583Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:57.4909022Z 2022-09-27T16:13:57.4909430Z ---------------------------------------------------------------------- 2022-09-27T16:13:57.4909841Z Ran 1 test in 0.001s 2022-09-27T16:13:57.4909958Z 2022-09-27T16:13:57.4910053Z OK (skipped=1) 2022-09-27T16:13:57.4910188Z 2022-09-27T16:13:57.4910275Z Generating XML reports... 2022-09-27T16:13:57.4933823Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161357.xml 2022-09-27T16:13:58.8087248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:13:58.8087895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:13:58.8090204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:13:58.8090865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:13:58.9463891Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:13:58.9472879Z 2022-09-27T16:13:58.9473000Z Running tests... 2022-09-27T16:13:58.9473423Z ---------------------------------------------------------------------- 2022-09-27T16:13:58.9479634Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:13:58.9480031Z 2022-09-27T16:13:58.9480283Z ---------------------------------------------------------------------- 2022-09-27T16:13:58.9480534Z Ran 1 test in 0.001s 2022-09-27T16:13:58.9480649Z 2022-09-27T16:13:58.9480721Z OK (skipped=1) 2022-09-27T16:13:58.9480831Z 2022-09-27T16:13:58.9480916Z Generating XML reports... 2022-09-27T16:13:58.9505756Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161358.xml 2022-09-27T16:14:00.2351867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:00.2352268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:00.2355496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:00.2355929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:00.3725245Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:00.3734665Z 2022-09-27T16:14:00.3734788Z Running tests... 2022-09-27T16:14:00.3735341Z ---------------------------------------------------------------------- 2022-09-27T16:14:00.3741570Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:14:00.3742043Z 2022-09-27T16:14:00.3742423Z ---------------------------------------------------------------------- 2022-09-27T16:14:00.3742677Z Ran 1 test in 0.001s 2022-09-27T16:14:00.3742781Z 2022-09-27T16:14:00.3742855Z OK (skipped=1) 2022-09-27T16:14:00.3742964Z 2022-09-27T16:14:00.3743052Z Generating XML reports... 2022-09-27T16:14:00.3766436Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161400.xml 2022-09-27T16:14:01.6585770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:01.6586157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:01.6588437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:01.6588782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:01.7962839Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:01.7972747Z 2022-09-27T16:14:01.7972853Z Running tests... 2022-09-27T16:14:01.7973251Z ---------------------------------------------------------------------- 2022-09-27T16:14:01.7979730Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:01.7980034Z 2022-09-27T16:14:01.7980322Z ---------------------------------------------------------------------- 2022-09-27T16:14:01.7980593Z Ran 1 test in 0.001s 2022-09-27T16:14:01.7980714Z 2022-09-27T16:14:01.7980788Z OK (skipped=1) 2022-09-27T16:14:01.7980884Z 2022-09-27T16:14:01.7980969Z Generating XML reports... 2022-09-27T16:14:01.8005321Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161401.xml 2022-09-27T16:14:03.1257300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:03.1257679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:03.1259821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:03.1260163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:03.2637112Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:03.2646868Z 2022-09-27T16:14:03.2646965Z Running tests... 2022-09-27T16:14:03.2647403Z ---------------------------------------------------------------------- 2022-09-27T16:14:03.2652960Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:03.2653309Z 2022-09-27T16:14:03.2653607Z ---------------------------------------------------------------------- 2022-09-27T16:14:03.2653962Z Ran 1 test in 0.001s 2022-09-27T16:14:03.2654076Z 2022-09-27T16:14:03.2654149Z OK (skipped=1) 2022-09-27T16:14:03.2654257Z 2022-09-27T16:14:03.2654335Z Generating XML reports... 2022-09-27T16:14:03.2678883Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161403.xml 2022-09-27T16:14:04.5889451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:04.5889812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:04.5891910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:04.5892432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:04.7267154Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:04.7277290Z 2022-09-27T16:14:04.7277410Z Running tests... 2022-09-27T16:14:04.7277872Z ---------------------------------------------------------------------- 2022-09-27T16:14:04.7282932Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:14:04.7283468Z 2022-09-27T16:14:04.7283878Z ---------------------------------------------------------------------- 2022-09-27T16:14:04.7284301Z Ran 1 test in 0.000s 2022-09-27T16:14:04.7284499Z 2022-09-27T16:14:04.7284641Z OK (skipped=1) 2022-09-27T16:14:04.7284840Z 2022-09-27T16:14:04.7284998Z Generating XML reports... 2022-09-27T16:14:04.7309094Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161404.xml 2022-09-27T16:14:06.0308472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:06.0308844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:06.0311281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:06.0311655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:06.1683216Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:06.1692557Z 2022-09-27T16:14:06.1692662Z Running tests... 2022-09-27T16:14:06.1693059Z ---------------------------------------------------------------------- 2022-09-27T16:14:06.1697440Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:14:06.1697871Z 2022-09-27T16:14:06.1698609Z ---------------------------------------------------------------------- 2022-09-27T16:14:06.1699053Z Ran 1 test in 0.000s 2022-09-27T16:14:06.1699221Z 2022-09-27T16:14:06.1699295Z OK (skipped=1) 2022-09-27T16:14:06.1699401Z 2022-09-27T16:14:06.1699760Z Generating XML reports... 2022-09-27T16:14:06.1725631Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161406.xml 2022-09-27T16:14:07.4728783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:07.4729146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:07.4731747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:07.4732106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:07.6090337Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:07.6099810Z 2022-09-27T16:14:07.6099950Z Running tests... 2022-09-27T16:14:07.6100400Z ---------------------------------------------------------------------- 2022-09-27T16:14:07.6113869Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:07.6114417Z 2022-09-27T16:14:07.6115153Z ---------------------------------------------------------------------- 2022-09-27T16:14:07.6115403Z Ran 1 test in 0.001s 2022-09-27T16:14:07.6115520Z 2022-09-27T16:14:07.6115593Z OK (skipped=1) 2022-09-27T16:14:07.6115710Z 2022-09-27T16:14:07.6115797Z Generating XML reports... 2022-09-27T16:14:07.6139883Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161407.xml 2022-09-27T16:14:08.9151263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:08.9151919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:08.9155135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:08.9155732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:09.0535523Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:09.0545567Z 2022-09-27T16:14:09.0545972Z Running tests... 2022-09-27T16:14:09.0546536Z ---------------------------------------------------------------------- 2022-09-27T16:14:09.0562886Z test_no_grad (__main__.DistributedDataParallelTest) 2022-09-27T16:14:09.0564342Z Note: this test can be sped up by only running it on a CPU module ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:14:09.0564728Z 2022-09-27T16:14:09.0565111Z ---------------------------------------------------------------------- 2022-09-27T16:14:09.0565513Z Ran 1 test in 0.002s 2022-09-27T16:14:09.0565697Z 2022-09-27T16:14:09.0565814Z OK (skipped=1) 2022-09-27T16:14:09.0565972Z 2022-09-27T16:14:09.0566113Z Generating XML reports... 2022-09-27T16:14:09.0590396Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161409.xml 2022-09-27T16:14:10.3609410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:10.3610046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:10.3612486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:10.3612873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:10.4986463Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:10.4996018Z 2022-09-27T16:14:10.4996139Z Running tests... 2022-09-27T16:14:10.4996600Z ---------------------------------------------------------------------- 2022-09-27T16:14:10.5007367Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:10.5008159Z 2022-09-27T16:14:10.5008429Z ---------------------------------------------------------------------- 2022-09-27T16:14:10.5008760Z Ran 1 test in 0.001s 2022-09-27T16:14:10.5008862Z 2022-09-27T16:14:10.5008935Z OK (skipped=1) 2022-09-27T16:14:10.5009045Z 2022-09-27T16:14:10.5009134Z Generating XML reports... 2022-09-27T16:14:10.5033410Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161410.xml 2022-09-27T16:14:11.7958931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:11.7959304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:11.7961977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:11.7962340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:11.9324663Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:11.9334524Z 2022-09-27T16:14:11.9334623Z Running tests... 2022-09-27T16:14:11.9335222Z ---------------------------------------------------------------------- 2022-09-27T16:14:11.9340804Z test_pass_default_pg (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:11.9341086Z 2022-09-27T16:14:11.9341442Z ---------------------------------------------------------------------- 2022-09-27T16:14:11.9341884Z Ran 1 test in 0.001s 2022-09-27T16:14:11.9342077Z 2022-09-27T16:14:11.9342160Z OK (skipped=1) 2022-09-27T16:14:11.9342269Z 2022-09-27T16:14:11.9342354Z Generating XML reports... 2022-09-27T16:14:11.9367556Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161411.xml 2022-09-27T16:14:13.2209495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:13.2209871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:13.2212155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:13.2212503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:13.3567781Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:13.3577488Z 2022-09-27T16:14:13.3577602Z Running tests... 2022-09-27T16:14:13.3578065Z ---------------------------------------------------------------------- 2022-09-27T16:14:13.3582572Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:14:13.3583035Z 2022-09-27T16:14:13.3583439Z ---------------------------------------------------------------------- 2022-09-27T16:14:13.3583866Z Ran 1 test in 0.000s 2022-09-27T16:14:13.3584088Z 2022-09-27T16:14:13.3584220Z OK (skipped=1) 2022-09-27T16:14:13.3584400Z 2022-09-27T16:14:13.3584555Z Generating XML reports... 2022-09-27T16:14:13.3608558Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161413.xml 2022-09-27T16:14:14.6436065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:14.6436731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:14.6439252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:14.6439802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:14.7811701Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:14.7821144Z 2022-09-27T16:14:14.7821466Z Running tests... 2022-09-27T16:14:14.7822389Z ---------------------------------------------------------------------- 2022-09-27T16:14:14.7826586Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:14:14.7827142Z 2022-09-27T16:14:14.7827392Z ---------------------------------------------------------------------- 2022-09-27T16:14:14.7827706Z Ran 1 test in 0.001s 2022-09-27T16:14:14.7827824Z 2022-09-27T16:14:14.7827902Z OK (skipped=1) 2022-09-27T16:14:14.7828014Z 2022-09-27T16:14:14.7828101Z Generating XML reports... 2022-09-27T16:14:14.7851555Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161414.xml 2022-09-27T16:14:16.0779309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:16.0779692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:16.0782279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:16.0782630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:16.2159018Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:16.2168162Z 2022-09-27T16:14:16.2168279Z Running tests... 2022-09-27T16:14:16.2168856Z ---------------------------------------------------------------------- 2022-09-27T16:14:16.4624668Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9534 2022-09-27T16:14:16.4650461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9535 2022-09-27T16:14:17.5492339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:17.5492983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:17.5496021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:17.5496604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:17.5817919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:17.5818558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:17.5821483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:17.5822108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:17.6853685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:14:17.7166218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:14:17.9684911Z skip: Need at least 2 CUDA devices (1.751s) 2022-09-27T16:14:17.9685134Z 2022-09-27T16:14:17.9685727Z ---------------------------------------------------------------------- 2022-09-27T16:14:17.9686189Z Ran 1 test in 1.752s 2022-09-27T16:14:17.9686392Z 2022-09-27T16:14:17.9686504Z OK (skipped=1) 2022-09-27T16:14:17.9686615Z 2022-09-27T16:14:17.9686710Z Generating XML reports... 2022-09-27T16:14:17.9720842Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161416.xml 2022-09-27T16:14:19.3853701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:19.3854155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:19.3856562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:19.3856924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:19.5233268Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:19.5243119Z 2022-09-27T16:14:19.5243372Z Running tests... 2022-09-27T16:14:19.5243968Z ---------------------------------------------------------------------- 2022-09-27T16:14:19.7746466Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9567 2022-09-27T16:14:19.7772059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9568 2022-09-27T16:14:20.8529353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:20.8529937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:20.8532178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:20.8532766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:20.8921707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:20.8922371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:20.8925358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:20.8925982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:20.9878296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:14:21.0266722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:14:21.2807504Z skip: Need at least 2 CUDA devices (1.756s) 2022-09-27T16:14:21.2807734Z 2022-09-27T16:14:21.2808081Z ---------------------------------------------------------------------- 2022-09-27T16:14:21.2808336Z Ran 1 test in 1.756s 2022-09-27T16:14:21.2808468Z 2022-09-27T16:14:21.2808545Z OK (skipped=1) 2022-09-27T16:14:21.2808643Z 2022-09-27T16:14:21.2808737Z Generating XML reports... 2022-09-27T16:14:21.2843800Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161419.xml 2022-09-27T16:14:22.7075674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:22.7076044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:22.7078587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:22.7079020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:22.8441431Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:22.8450981Z 2022-09-27T16:14:22.8451058Z Running tests... 2022-09-27T16:14:22.8452220Z ---------------------------------------------------------------------- 2022-09-27T16:14:22.8456870Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:14:22.8457167Z 2022-09-27T16:14:22.8457557Z ---------------------------------------------------------------------- 2022-09-27T16:14:22.8457998Z Ran 1 test in 0.001s 2022-09-27T16:14:22.8458106Z 2022-09-27T16:14:22.8458178Z OK (skipped=1) 2022-09-27T16:14:22.8458285Z 2022-09-27T16:14:22.8458369Z Generating XML reports... 2022-09-27T16:14:22.8483245Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161422.xml 2022-09-27T16:14:24.1723911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:24.1724290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:24.1727065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:24.1727416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:24.3093860Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:24.3103234Z 2022-09-27T16:14:24.3103372Z Running tests... 2022-09-27T16:14:24.3103973Z ---------------------------------------------------------------------- 2022-09-27T16:14:24.3112098Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:24.3112519Z 2022-09-27T16:14:24.3112803Z ---------------------------------------------------------------------- 2022-09-27T16:14:24.3113053Z Ran 1 test in 0.001s 2022-09-27T16:14:24.3113170Z 2022-09-27T16:14:24.3113247Z OK (skipped=1) 2022-09-27T16:14:24.3113355Z 2022-09-27T16:14:24.3113433Z Generating XML reports... 2022-09-27T16:14:24.3137222Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161424.xml 2022-09-27T16:14:25.6228419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:25.6246597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:25.6247416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:25.6247960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:25.7616637Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:25.7625881Z 2022-09-27T16:14:25.7626180Z Running tests... 2022-09-27T16:14:25.7626646Z ---------------------------------------------------------------------- 2022-09-27T16:14:25.7632125Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:14:25.7632548Z 2022-09-27T16:14:25.7632920Z ---------------------------------------------------------------------- 2022-09-27T16:14:25.7633256Z Ran 1 test in 0.001s 2022-09-27T16:14:25.7633376Z 2022-09-27T16:14:25.7633457Z OK (skipped=1) 2022-09-27T16:14:25.7633569Z 2022-09-27T16:14:25.7633644Z Generating XML reports... 2022-09-27T16:14:25.7658019Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161425.xml 2022-09-27T16:14:27.0595245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:27.0595626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:27.0598236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:27.0598603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:27.1963572Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:27.1973397Z 2022-09-27T16:14:27.1973505Z Running tests... 2022-09-27T16:14:27.1974246Z ---------------------------------------------------------------------- 2022-09-27T16:14:27.1979111Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:14:27.1979538Z 2022-09-27T16:14:27.1980014Z ---------------------------------------------------------------------- 2022-09-27T16:14:27.1980498Z Ran 1 test in 0.001s 2022-09-27T16:14:27.1980696Z 2022-09-27T16:14:27.1980775Z OK (skipped=1) 2022-09-27T16:14:27.1980890Z 2022-09-27T16:14:27.1980977Z Generating XML reports... 2022-09-27T16:14:27.2005281Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161427.xml 2022-09-27T16:14:28.4906295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:28.4906666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:28.4908378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:28.4908736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:28.6269461Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:28.6279585Z 2022-09-27T16:14:28.6279857Z Running tests... 2022-09-27T16:14:28.6280499Z ---------------------------------------------------------------------- 2022-09-27T16:14:28.6285402Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:14:28.6285826Z 2022-09-27T16:14:28.6286240Z ---------------------------------------------------------------------- 2022-09-27T16:14:28.6286686Z Ran 1 test in 0.001s 2022-09-27T16:14:28.6286907Z 2022-09-27T16:14:28.6287041Z OK (skipped=1) 2022-09-27T16:14:28.6287233Z 2022-09-27T16:14:28.6287369Z Generating XML reports... 2022-09-27T16:14:28.6310185Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161428.xml 2022-09-27T16:14:29.9164460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:29.9164857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:29.9167316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:29.9167680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:30.0532431Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:30.0542650Z 2022-09-27T16:14:30.0542863Z Running tests... 2022-09-27T16:14:30.0543450Z ---------------------------------------------------------------------- 2022-09-27T16:14:30.0548231Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:14:30.0548608Z 2022-09-27T16:14:30.0548882Z ---------------------------------------------------------------------- 2022-09-27T16:14:30.0549133Z Ran 1 test in 0.001s 2022-09-27T16:14:30.0549247Z 2022-09-27T16:14:30.0549326Z OK (skipped=1) 2022-09-27T16:14:30.0549419Z 2022-09-27T16:14:30.0549504Z Generating XML reports... 2022-09-27T16:14:30.0574936Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161430.xml 2022-09-27T16:14:31.3497363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:31.3497974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:31.3500193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:31.3500757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:31.4866390Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:31.4876580Z 2022-09-27T16:14:31.4877040Z Running tests... 2022-09-27T16:14:31.4877433Z ---------------------------------------------------------------------- 2022-09-27T16:14:31.4882853Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:14:31.4883277Z 2022-09-27T16:14:31.4883723Z ---------------------------------------------------------------------- 2022-09-27T16:14:31.4883987Z Ran 1 test in 0.001s 2022-09-27T16:14:31.4884103Z 2022-09-27T16:14:31.4884164Z OK (skipped=1) 2022-09-27T16:14:31.4884273Z 2022-09-27T16:14:31.4884358Z Generating XML reports... 2022-09-27T16:14:31.4907388Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161431.xml 2022-09-27T16:14:32.7743212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:32.7743868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:32.7746006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:32.7746365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:32.9126723Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:32.9136243Z 2022-09-27T16:14:32.9136347Z Running tests... 2022-09-27T16:14:32.9136728Z ---------------------------------------------------------------------- 2022-09-27T16:14:32.9149235Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:32.9149658Z 2022-09-27T16:14:32.9150061Z ---------------------------------------------------------------------- 2022-09-27T16:14:32.9150477Z Ran 1 test in 0.001s 2022-09-27T16:14:32.9150764Z 2022-09-27T16:14:32.9150878Z OK (skipped=1) 2022-09-27T16:14:32.9151070Z 2022-09-27T16:14:32.9151227Z Generating XML reports... 2022-09-27T16:14:32.9175310Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161432.xml 2022-09-27T16:14:34.2178521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:34.2178906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:34.2181081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:34.2181445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:34.3553876Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:34.3563933Z 2022-09-27T16:14:34.3564149Z Running tests... 2022-09-27T16:14:34.3564482Z ---------------------------------------------------------------------- 2022-09-27T16:14:34.3576180Z test_nccl_timeout (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:34.3576441Z 2022-09-27T16:14:34.3576775Z ---------------------------------------------------------------------- 2022-09-27T16:14:34.3577064Z Ran 1 test in 0.001s 2022-09-27T16:14:34.3577164Z 2022-09-27T16:14:34.3577242Z OK (skipped=1) 2022-09-27T16:14:34.3577379Z 2022-09-27T16:14:34.3577498Z Generating XML reports... 2022-09-27T16:14:34.3602150Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161434.xml 2022-09-27T16:14:35.6531317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:35.6531695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:35.6534387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:35.6534767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:35.7890570Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:35.7899511Z 2022-09-27T16:14:35.7899646Z Running tests... 2022-09-27T16:14:35.7900222Z ---------------------------------------------------------------------- 2022-09-27T16:14:35.7906233Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:35.7906643Z 2022-09-27T16:14:35.7907258Z ---------------------------------------------------------------------- 2022-09-27T16:14:35.7907701Z Ran 1 test in 0.001s 2022-09-27T16:14:35.7907829Z 2022-09-27T16:14:35.7908134Z OK (skipped=1) 2022-09-27T16:14:35.7908253Z 2022-09-27T16:14:35.7908340Z Generating XML reports... 2022-09-27T16:14:35.7932118Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220927161435.xml 2022-09-27T16:14:37.0763106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:37.0763485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:37.0765244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:37.0765743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:37.2129884Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:37.2139342Z 2022-09-27T16:14:37.2139470Z Running tests... 2022-09-27T16:14:37.2140059Z ---------------------------------------------------------------------- 2022-09-27T16:14:37.2151919Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:37.2152460Z 2022-09-27T16:14:37.2152815Z ---------------------------------------------------------------------- 2022-09-27T16:14:37.2153270Z Ran 1 test in 0.001s 2022-09-27T16:14:37.2153484Z 2022-09-27T16:14:37.2153613Z OK (skipped=1) 2022-09-27T16:14:37.2153738Z 2022-09-27T16:14:37.2153812Z Generating XML reports... 2022-09-27T16:14:37.2177214Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161437.xml 2022-09-27T16:14:38.5013643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:38.5014027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:38.5016549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:38.5016915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:38.6386500Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:38.6396150Z 2022-09-27T16:14:38.6396301Z Running tests... 2022-09-27T16:14:38.6396716Z ---------------------------------------------------------------------- 2022-09-27T16:14:38.6406173Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:38.6406583Z 2022-09-27T16:14:38.6406982Z ---------------------------------------------------------------------- 2022-09-27T16:14:38.6407253Z Ran 1 test in 0.001s 2022-09-27T16:14:38.6407371Z 2022-09-27T16:14:38.6407432Z OK (skipped=1) 2022-09-27T16:14:38.6407543Z 2022-09-27T16:14:38.6407629Z Generating XML reports... 2022-09-27T16:14:38.6431013Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161438.xml 2022-09-27T16:14:39.9240269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:39.9243533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:39.9244329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:39.9244940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:40.0621651Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:40.0630855Z 2022-09-27T16:14:40.0631228Z Running tests... 2022-09-27T16:14:40.0645608Z ---------------------------------------------------------------------- 2022-09-27T16:14:40.0646315Z test_allgather_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:40.0646603Z 2022-09-27T16:14:40.0647083Z ---------------------------------------------------------------------- 2022-09-27T16:14:40.0647334Z Ran 1 test in 0.001s 2022-09-27T16:14:40.0647533Z 2022-09-27T16:14:40.0647609Z OK (skipped=1) 2022-09-27T16:14:40.0647704Z 2022-09-27T16:14:40.0647790Z Generating XML reports... 2022-09-27T16:14:40.0670624Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161440.xml 2022-09-27T16:14:41.3490797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:41.3491207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:41.3493750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:41.3494117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:41.4853973Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:41.4863555Z 2022-09-27T16:14:41.4863663Z Running tests... 2022-09-27T16:14:41.4864499Z ---------------------------------------------------------------------- 2022-09-27T16:14:41.4892916Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.003s) 2022-09-27T16:14:41.4893229Z 2022-09-27T16:14:41.4893845Z ---------------------------------------------------------------------- 2022-09-27T16:14:41.4894168Z Ran 1 test in 0.003s 2022-09-27T16:14:41.4894329Z 2022-09-27T16:14:41.4894411Z OK (skipped=1) 2022-09-27T16:14:41.4894521Z 2022-09-27T16:14:41.4894609Z Generating XML reports... 2022-09-27T16:14:41.4918236Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161441.xml 2022-09-27T16:14:42.7659686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:42.7660132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:42.7662619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:42.7662992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:42.9038822Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:42.9048333Z 2022-09-27T16:14:42.9048468Z Running tests... 2022-09-27T16:14:42.9049060Z ---------------------------------------------------------------------- 2022-09-27T16:14:42.9062347Z test_barrier (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:42.9062668Z 2022-09-27T16:14:42.9062928Z ---------------------------------------------------------------------- 2022-09-27T16:14:42.9063256Z Ran 1 test in 0.001s 2022-09-27T16:14:42.9063460Z 2022-09-27T16:14:42.9063591Z OK (skipped=1) 2022-09-27T16:14:42.9063785Z 2022-09-27T16:14:42.9063954Z Generating XML reports... 2022-09-27T16:14:42.9087923Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161442.xml 2022-09-27T16:14:44.1998811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:44.1999197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:44.2002072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:44.2002432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:44.3374362Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:44.3384279Z 2022-09-27T16:14:44.3384366Z Running tests... 2022-09-27T16:14:44.3385086Z ---------------------------------------------------------------------- 2022-09-27T16:14:44.3401022Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:44.3401490Z 2022-09-27T16:14:44.3401881Z ---------------------------------------------------------------------- 2022-09-27T16:14:44.3402133Z Ran 1 test in 0.002s 2022-09-27T16:14:44.3402252Z 2022-09-27T16:14:44.3402313Z OK (skipped=1) 2022-09-27T16:14:44.3402425Z 2022-09-27T16:14:44.3402510Z Generating XML reports... 2022-09-27T16:14:44.3426927Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161444.xml 2022-09-27T16:14:45.6381900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:45.6382276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:45.6384674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:45.6385054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:45.7744058Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:45.7754181Z 2022-09-27T16:14:45.7754273Z Running tests... 2022-09-27T16:14:45.7755199Z ---------------------------------------------------------------------- 2022-09-27T16:14:45.7770643Z test_empty_tensors (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:45.7771049Z 2022-09-27T16:14:45.7771437Z ---------------------------------------------------------------------- 2022-09-27T16:14:45.7771814Z Ran 1 test in 0.002s 2022-09-27T16:14:45.7772004Z 2022-09-27T16:14:45.7772123Z OK (skipped=1) 2022-09-27T16:14:45.7772309Z 2022-09-27T16:14:45.7772459Z Generating XML reports... 2022-09-27T16:14:45.7796450Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161445.xml 2022-09-27T16:14:47.0741084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:47.0741456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:47.0744368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:47.0744716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:47.2107196Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:47.2116572Z 2022-09-27T16:14:47.2116883Z Running tests... 2022-09-27T16:14:47.2117543Z ---------------------------------------------------------------------- 2022-09-27T16:14:47.2136817Z test_gather_checks (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:14:47.2137708Z 2022-09-27T16:14:47.2138087Z ---------------------------------------------------------------------- 2022-09-27T16:14:47.2138528Z Ran 1 test in 0.002s 2022-09-27T16:14:47.2138741Z 2022-09-27T16:14:47.2138873Z OK (skipped=1) 2022-09-27T16:14:47.2139057Z 2022-09-27T16:14:47.2139138Z Generating XML reports... 2022-09-27T16:14:47.2161444Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161447.xml 2022-09-27T16:14:48.4891003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:48.4891405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:48.4891914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:48.4892325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:48.6246319Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:48.6255510Z 2022-09-27T16:14:48.6255658Z Running tests... 2022-09-27T16:14:48.6256336Z ---------------------------------------------------------------------- 2022-09-27T16:14:48.6270596Z test_gather_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:48.6271263Z 2022-09-27T16:14:48.6271784Z ---------------------------------------------------------------------- 2022-09-27T16:14:48.6272278Z Ran 1 test in 0.002s 2022-09-27T16:14:48.6272487Z 2022-09-27T16:14:48.6272560Z OK (skipped=1) 2022-09-27T16:14:48.6272669Z 2022-09-27T16:14:48.6272753Z Generating XML reports... 2022-09-27T16:14:48.6296261Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161448.xml 2022-09-27T16:14:49.9238640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:49.9239016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:49.9242013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:49.9242383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:50.0603440Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:50.0612303Z 2022-09-27T16:14:50.0612525Z Running tests... 2022-09-27T16:14:50.0612951Z ---------------------------------------------------------------------- 2022-09-27T16:14:50.0629492Z test_gather_stress (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:14:50.0629754Z 2022-09-27T16:14:50.0630115Z ---------------------------------------------------------------------- 2022-09-27T16:14:50.0630565Z Ran 1 test in 0.002s 2022-09-27T16:14:50.0630763Z 2022-09-27T16:14:50.0630845Z OK (skipped=1) 2022-09-27T16:14:50.0630958Z 2022-09-27T16:14:50.0631044Z Generating XML reports... 2022-09-27T16:14:50.0654230Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161450.xml 2022-09-27T16:14:51.3512949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:51.3513327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:51.3516254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:51.3516618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:51.4862544Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:51.4872289Z 2022-09-27T16:14:51.4872398Z Running tests... 2022-09-27T16:14:51.4872985Z ---------------------------------------------------------------------- 2022-09-27T16:14:51.4895215Z test_reduce_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:14:51.4895593Z 2022-09-27T16:14:51.4895880Z ---------------------------------------------------------------------- 2022-09-27T16:14:51.4896235Z Ran 1 test in 0.002s 2022-09-27T16:14:51.4896359Z 2022-09-27T16:14:51.4896437Z OK (skipped=1) 2022-09-27T16:14:51.4896544Z 2022-09-27T16:14:51.4896629Z Generating XML reports... 2022-09-27T16:14:51.4920234Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161451.xml 2022-09-27T16:14:52.7533984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:52.7534366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:52.7536638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:52.7537187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:52.8904076Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:52.8913248Z 2022-09-27T16:14:52.8913391Z Running tests... 2022-09-27T16:14:52.8914142Z ---------------------------------------------------------------------- 2022-09-27T16:14:52.8926765Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:52.8927209Z 2022-09-27T16:14:52.8927613Z ---------------------------------------------------------------------- 2022-09-27T16:14:52.8927865Z Ran 1 test in 0.001s 2022-09-27T16:14:52.8927967Z 2022-09-27T16:14:52.8928040Z OK (skipped=1) 2022-09-27T16:14:52.8928148Z 2022-09-27T16:14:52.8928234Z Generating XML reports... 2022-09-27T16:14:52.8952113Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161452.xml 2022-09-27T16:14:54.1884162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:54.1884541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:54.1886784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:54.1887302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:54.3253123Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:54.3262349Z 2022-09-27T16:14:54.3262497Z Running tests... 2022-09-27T16:14:54.3272676Z ---------------------------------------------------------------------- 2022-09-27T16:14:54.3273342Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:54.3273574Z 2022-09-27T16:14:54.3273795Z ---------------------------------------------------------------------- 2022-09-27T16:14:54.3274038Z Ran 1 test in 0.001s 2022-09-27T16:14:54.3274142Z 2022-09-27T16:14:54.3274226Z OK (skipped=1) 2022-09-27T16:14:54.3274334Z 2022-09-27T16:14:54.3274419Z Generating XML reports... 2022-09-27T16:14:54.3297149Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161454.xml 2022-09-27T16:14:55.5978280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:55.5978662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:55.5981178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:55.5981544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:55.7332498Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:55.7341653Z 2022-09-27T16:14:55.7341789Z Running tests... 2022-09-27T16:14:55.7342809Z ---------------------------------------------------------------------- 2022-09-27T16:14:55.7381493Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.004s) 2022-09-27T16:14:55.7381887Z 2022-09-27T16:14:55.7382406Z ---------------------------------------------------------------------- 2022-09-27T16:14:55.7382861Z Ran 1 test in 0.004s 2022-09-27T16:14:55.7383043Z 2022-09-27T16:14:55.7383150Z OK (skipped=1) 2022-09-27T16:14:55.7383245Z 2022-09-27T16:14:55.7383331Z Generating XML reports... 2022-09-27T16:14:55.7406982Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161455.xml 2022-09-27T16:14:57.0361379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:57.0361761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:57.0363711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:57.0364076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:57.1753792Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:57.1764139Z 2022-09-27T16:14:57.1764395Z Running tests... 2022-09-27T16:14:57.1764966Z ---------------------------------------------------------------------- 2022-09-27T16:14:57.1779985Z test_scatter_checks (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:57.1780267Z 2022-09-27T16:14:57.1780534Z ---------------------------------------------------------------------- 2022-09-27T16:14:57.1780766Z Ran 1 test in 0.002s 2022-09-27T16:14:57.1780881Z 2022-09-27T16:14:57.1780956Z OK (skipped=1) 2022-09-27T16:14:57.1781066Z 2022-09-27T16:14:57.1781152Z Generating XML reports... 2022-09-27T16:14:57.1806252Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161457.xml 2022-09-27T16:14:58.4707173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:58.4707554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:58.4709756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:58.4710097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:14:58.6071609Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:14:58.6081130Z 2022-09-27T16:14:58.6081261Z Running tests... 2022-09-27T16:14:58.6081865Z ---------------------------------------------------------------------- 2022-09-27T16:14:58.6096282Z test_scatter_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:14:58.6096636Z 2022-09-27T16:14:58.6096939Z ---------------------------------------------------------------------- 2022-09-27T16:14:58.6097193Z Ran 1 test in 0.002s 2022-09-27T16:14:58.6097297Z 2022-09-27T16:14:58.6097379Z OK (skipped=1) 2022-09-27T16:14:58.6097487Z 2022-09-27T16:14:58.6097573Z Generating XML reports... 2022-09-27T16:14:58.6121562Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161458.xml 2022-09-27T16:14:59.9117933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:14:59.9118302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:14:59.9121198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:14:59.9121584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:00.0494206Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:15:00.0503452Z 2022-09-27T16:15:00.0503527Z Running tests... 2022-09-27T16:15:00.0504549Z ---------------------------------------------------------------------- 2022-09-27T16:15:00.0520893Z test_scatter_stress (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:15:00.0521350Z 2022-09-27T16:15:00.0521809Z ---------------------------------------------------------------------- 2022-09-27T16:15:00.0522281Z Ran 1 test in 0.002s 2022-09-27T16:15:00.0522435Z 2022-09-27T16:15:00.0522515Z OK (skipped=1) 2022-09-27T16:15:00.0522622Z 2022-09-27T16:15:00.0522708Z Generating XML reports... 2022-09-27T16:15:00.0546789Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161500.xml 2022-09-27T16:15:01.3395511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:01.3396112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:01.3398158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:01.3398673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:01.4747473Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:15:01.4757176Z 2022-09-27T16:15:01.4757285Z Running tests... 2022-09-27T16:15:01.4757745Z ---------------------------------------------------------------------- 2022-09-27T16:15:01.4768441Z test_send_recv (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:15:01.4768767Z 2022-09-27T16:15:01.4769033Z ---------------------------------------------------------------------- 2022-09-27T16:15:01.4769274Z Ran 1 test in 0.001s 2022-09-27T16:15:01.4769387Z 2022-09-27T16:15:01.4769458Z OK (skipped=1) 2022-09-27T16:15:01.4769566Z 2022-09-27T16:15:01.4769658Z Generating XML reports... 2022-09-27T16:15:01.4793139Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161501.xml 2022-09-27T16:15:02.7737568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:02.7737949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:02.7740538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:02.7740896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:02.9100789Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:15:02.9110522Z 2022-09-27T16:15:02.9110947Z Running tests... 2022-09-27T16:15:02.9111351Z ---------------------------------------------------------------------- 2022-09-27T16:15:02.9146571Z test_common_errors (__main__.RendezvousEnvTest) ... skip: c10d was not compiled with the NCCL backend (0.003s) 2022-09-27T16:15:02.9146909Z 2022-09-27T16:15:02.9147295Z ---------------------------------------------------------------------- 2022-09-27T16:15:02.9147746Z Ran 1 test in 0.004s 2022-09-27T16:15:02.9147950Z 2022-09-27T16:15:02.9148078Z OK (skipped=1) 2022-09-27T16:15:02.9148271Z 2022-09-27T16:15:02.9148420Z Generating XML reports... 2022-09-27T16:15:02.9172122Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220927161502.xml 2022-09-27T16:15:04.2026856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:04.2027245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:04.2030414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:04.2030954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:04.3400652Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-09-27T16:15:04.3409903Z 2022-09-27T16:15:04.3410029Z Running tests... 2022-09-27T16:15:04.3410620Z ---------------------------------------------------------------------- 2022-09-27T16:15:04.3415254Z test_default_store_timeout_nccl (__main__.TimeoutTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:15:04.3415612Z 2022-09-27T16:15:04.3415959Z ---------------------------------------------------------------------- 2022-09-27T16:15:04.3416209Z Ran 1 test in 0.001s 2022-09-27T16:15:04.3416326Z 2022-09-27T16:15:04.3416403Z OK (skipped=1) 2022-09-27T16:15:04.3416497Z 2022-09-27T16:15:04.3416584Z Generating XML reports... 2022-09-27T16:15:04.3440476Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220927161504.xml 2022-09-27T16:15:04.7848046Z Running distributed/fsdp/test_wrap ... [2022-09-27 16:15:04.784410] 2022-09-27T16:15:04.7848632Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:15:04.784494] 2022-09-27T16:15:06.0104766Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap 2022-09-27T16:15:06.0120043Z 2022-09-27T16:15:06.0120177Z Running tests... 2022-09-27T16:15:06.0120621Z ---------------------------------------------------------------------- 2022-09-27T16:15:06.0127009Z test_always_wrap (__main__.TestAutoWrap) 2022-09-27T16:15:06.0127657Z Test to ensure that if `always_wrap_policy` is ... skip: Test Requires CUDA (0.001s) 2022-09-27T16:15:06.0139431Z test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-09-27T16:15:06.0150530Z test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-09-27T16:15:06.0155400Z test_auto_wrap_api (__main__.TestAutoWrap) 2022-09-27T16:15:06.0156720Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... skip: Requires at least 2 GPUs (0.000s) 2022-09-27T16:15:06.0163007Z test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap) 2022-09-27T16:15:06.0163372Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... skip: Requires at least 2 GPUs (0.001s) 2022-09-27T16:15:06.0168712Z test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap) 2022-09-27T16:15:06.0169594Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... skip: Requires at least 2 GPUs (0.001s) 2022-09-27T16:15:06.0176869Z test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap) 2022-09-27T16:15:06.0177777Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... skip: Requires at least 2 GPUs (0.001s) 2022-09-27T16:15:06.0185578Z test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap) 2022-09-27T16:15:06.0186342Z Test to ensure force-leaf modules are not wrapped. ... skip: Requires at least 2 GPUs (0.001s) 2022-09-27T16:15:06.0202858Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s) 2022-09-27T16:15:06.0218763Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s) 2022-09-27T16:15:06.0233849Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s) 2022-09-27T16:15:06.0249494Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s) 2022-09-27T16:15:06.0265052Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s) 2022-09-27T16:15:06.0280606Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s) 2022-09-27T16:15:06.0295978Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s) 2022-09-27T16:15:06.0311404Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s) 2022-09-27T16:15:06.0323707Z test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-09-27T16:15:06.0335284Z test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-09-27T16:15:06.0342789Z test_transformer_auto_wrap_policy (__main__.TestAutoWrap) 2022-09-27T16:15:06.0343160Z Tests the ``transformer_auto_wrap_policy``. ... skip: Requires at least 2 GPUs (0.001s) 2022-09-27T16:15:06.0350652Z test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-09-27T16:15:06.0357169Z test_wrap_override_defaults (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-09-27T16:15:06.0365448Z test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-09-27T16:15:06.0373663Z test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-09-27T16:15:06.0384755Z test_bn_always_wrapped_individually (__main__.TestFSDPWrap) 2022-09-27T16:15:06.2929179Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9900 2022-09-27T16:15:06.2955787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9901 2022-09-27T16:15:06.2983003Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9902 2022-09-27T16:15:06.3012796Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9903 2022-09-27T16:15:07.5114836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:07.5115464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:07.5118855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:07.5119489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:07.5227573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:07.5228122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:07.5231854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:07.5232511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:07.6268666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:07.6269284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:07.6272264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:07.6272873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:07.6307042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:07.6307700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:07.6311992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:07.6312812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:07.6401860Z dist init r=3, world=4 2022-09-27T16:15:07.6518632Z dist init r=1, world=4 2022-09-27T16:15:07.7553577Z dist init r=2, world=4 2022-09-27T16:15:07.7578394Z dist init r=0, world=4 2022-09-27T16:15:07.7992875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:07.7993268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:07.8093945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:07.8094685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:07.8095712Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:07.8096233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:07.8096753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:07.8097274Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:07.8201486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:07.8202010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:07.8202543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:07.8203072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:08.1058904Z skip: Need at least 2 CUDA devices (2.068s) 2022-09-27T16:15:08.1066698Z test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) 2022-09-27T16:15:08.1111882Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9956 2022-09-27T16:15:08.1145905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9957 2022-09-27T16:15:08.1174550Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9958 2022-09-27T16:15:08.1203175Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9959 2022-09-27T16:15:09.2960443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:09.2961096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:09.2964822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:09.2965473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:09.3054642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:09.3055255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:09.3059588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:09.3060471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:09.3105652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:09.3106239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:09.3110453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:09.3111104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:09.3166652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:09.3167212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:09.3171898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:09.3172523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:09.4349281Z dist init r=1, world=4 2022-09-27T16:15:09.4411173Z dist init r=3, world=4 2022-09-27T16:15:09.4411701Z dist init r=0, world=4 2022-09-27T16:15:09.4494038Z dist init r=2, world=4 2022-09-27T16:15:09.4619125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:09.4720563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:09.4805403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:09.4805985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:09.4806852Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:09.4822436Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:09.4823059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:09.4823593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:09.4913116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:09.4913654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:09.4914213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:09.4914841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:09.8246184Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:15:09.8252810Z test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) 2022-09-27T16:15:09.8298312Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10012 2022-09-27T16:15:09.8332526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10013 2022-09-27T16:15:09.8361609Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10014 2022-09-27T16:15:09.8389858Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10015 2022-09-27T16:15:10.9841830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:10.9842451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:10.9845358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:10.9845953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:10.9879135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:10.9879803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:10.9880590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:10.9881188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:10.9884929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:10.9885960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:10.9887127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:10.9887730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:10.9921213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:10.9922091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:10.9925775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:10.9926569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:11.1140551Z dist init r=1, world=4 2022-09-27T16:15:11.1160775Z dist init r=3, world=4 2022-09-27T16:15:11.1168794Z dist init r=2, world=4 2022-09-27T16:15:11.1197816Z dist init r=0, world=4 2022-09-27T16:15:11.1471158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:11.1572678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:11.1675080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:11.1675814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:11.1676764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:11.1677423Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:11.1678101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:11.1678675Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:11.1682569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:11.1683110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:11.1683626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:11.1684164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:11.4432411Z skip: Need at least 2 CUDA devices (1.618s) 2022-09-27T16:15:11.4441501Z test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) 2022-09-27T16:15:11.4496523Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10068 2022-09-27T16:15:11.4533681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10069 2022-09-27T16:15:11.4562437Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10070 2022-09-27T16:15:11.4590340Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10071 2022-09-27T16:15:12.6482709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:12.6483222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:12.6486349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:12.6486855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:12.6851213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:12.6851869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:12.6855554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:12.6856192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:12.7019550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:12.7020161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:12.7023793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:12.7024436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:12.7060154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:12.7060805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:12.7065675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:12.7066295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:12.7770858Z dist init r=0, world=4 2022-09-27T16:15:12.8142686Z dist init r=2, world=4 2022-09-27T16:15:12.8296417Z dist init r=1, world=4 2022-09-27T16:15:12.8334277Z dist init r=3, world=4 2022-09-27T16:15:12.8443746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:12.8645588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:12.8748329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:12.8749189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:12.8750083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:12.8750873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:12.8751620Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:12.8752408Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:12.8756208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:12.8756991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:12.8757758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:12.8759183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:13.1632086Z skip: Need at least 2 CUDA devices (1.720s) 2022-09-27T16:15:13.1638989Z test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) 2022-09-27T16:15:13.1684235Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10124 2022-09-27T16:15:13.1718344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10125 2022-09-27T16:15:13.1746703Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10126 2022-09-27T16:15:13.1775367Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10127 2022-09-27T16:15:14.3124520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:14.3125045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:14.3128851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:14.3129361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:14.3362929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:14.3363649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:14.3366819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:14.3367792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:14.3598341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:14.3599274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:14.3600600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:14.3601205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:14.3781305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:14.3781968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:14.3785635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:14.3786266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:14.4404730Z dist init r=1, world=4 2022-09-27T16:15:14.4638211Z dist init r=0, world=4 2022-09-27T16:15:14.4861018Z dist init r=2, world=4 2022-09-27T16:15:14.5050995Z dist init r=3, world=4 2022-09-27T16:15:14.5170681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:14.5217376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:14.5218421Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:14.5218982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:14.5219387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:14.5219904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:14.5220434Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:14.5273138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:14.5326709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:14.5327383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:14.5328044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:14.5329007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:14.8816671Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T16:15:14.8886012Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10180 2022-09-27T16:15:14.8920085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10181 2022-09-27T16:15:14.8948227Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10182 2022-09-27T16:15:14.8975338Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10183 2022-09-27T16:15:16.0793437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:16.0794052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:16.0797168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:16.0797836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:16.0890878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:16.0891456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:16.0895059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:16.0895699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:16.1063253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:16.1063870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:16.1067735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:16.1068361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:16.1261912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:16.1262618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:16.1266765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:16.1268666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:16.2074184Z dist init r=0, world=4 2022-09-27T16:15:16.2179972Z dist init r=1, world=4 2022-09-27T16:15:16.2334882Z dist init r=3, world=4 2022-09-27T16:15:16.2526050Z dist init r=2, world=4 2022-09-27T16:15:16.2736287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:16.2837655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:16.2939337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:16.2939955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:16.2940749Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:16.2941534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:16.2942174Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:16.2942882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:16.2947021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:16.2947611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:16.2948145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:16.2948673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:16.6016490Z skip: Need at least 2 CUDA devices (1.720s) 2022-09-27T16:15:16.6086373Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10236 2022-09-27T16:15:16.6121751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10237 2022-09-27T16:15:16.6150328Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10238 2022-09-27T16:15:16.6187081Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10239 2022-09-27T16:15:17.7847066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:17.7848033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:17.7851003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:17.7851761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:17.7928156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:17.7928789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:17.7932912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:17.7933537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:17.8212317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:17.8212953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:17.8216011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:17.8216744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:17.8217500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:17.8218083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:17.8221454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:17.8222095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:17.9125612Z dist init r=2, world=4 2022-09-27T16:15:17.9203010Z dist init r=1, world=4 2022-09-27T16:15:17.9479086Z dist init r=3, world=4 2022-09-27T16:15:17.9486666Z dist init r=0, world=4 2022-09-27T16:15:17.9688247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:17.9788911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:17.9815140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:17.9815532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:17.9816140Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:17.9816673Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:17.9892433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:17.9893053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:17.9921899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:17.9922856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:17.9923383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:17.9923910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:18.3226592Z skip: Need at least 2 CUDA devices (1.721s) 2022-09-27T16:15:18.3296873Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10292 2022-09-27T16:15:18.3331064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10293 2022-09-27T16:15:18.3358650Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10294 2022-09-27T16:15:18.3386934Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10295 2022-09-27T16:15:19.4857085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:19.4857741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:19.4860770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:19.4861427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:19.4912194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:19.4912818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:19.4917125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:19.4917744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:19.4983043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:19.4983581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:19.4988262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:19.4988838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:19.5072606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:19.5073210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:19.5077585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:19.5078188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:19.6127335Z dist init r=0, world=4 2022-09-27T16:15:19.6191232Z dist init r=1, world=4 2022-09-27T16:15:19.6253312Z dist init r=3, world=4 2022-09-27T16:15:19.6333854Z dist init r=2, world=4 2022-09-27T16:15:19.6602589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:19.6603184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:19.6704745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:19.6705282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:19.6706073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:19.6706624Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:19.6707136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:19.6707658Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:19.6811708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:19.6812391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:19.6812900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:19.6813434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:20.0428375Z skip: Need at least 2 CUDA devices (1.720s) 2022-09-27T16:15:20.0497936Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10348 2022-09-27T16:15:20.0531165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10349 2022-09-27T16:15:20.0559343Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10350 2022-09-27T16:15:20.0588311Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10351 2022-09-27T16:15:21.2208390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:21.2208936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:21.2212133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:21.2212797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:21.2277784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:21.2278332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:21.2282251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:21.2282826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:21.2473127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:21.2473782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:21.2478188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:21.2478839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:21.2533021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:21.2533672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:21.2538427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:21.2539469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:21.3499347Z dist init r=1, world=4 2022-09-27T16:15:21.3561799Z dist init r=2, world=4 2022-09-27T16:15:21.3762406Z dist init r=0, world=4 2022-09-27T16:15:21.3804867Z dist init r=3, world=4 2022-09-27T16:15:21.4016700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:21.4117002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:21.4219478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:21.4220414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:21.4221055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:21.4221952Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:21.4222916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:21.4223584Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:21.4225758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:21.4226495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:21.4229703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:21.4231095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:21.7630565Z skip: Need at least 2 CUDA devices (1.720s) 2022-09-27T16:15:21.7700281Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10404 2022-09-27T16:15:21.7734969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10405 2022-09-27T16:15:21.7764215Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10406 2022-09-27T16:15:21.7792490Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10407 2022-09-27T16:15:22.8875884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:22.8876650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:22.8878764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:22.8879375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:22.9193827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:22.9194669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:22.9198140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:22.9198830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:22.9315378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:22.9316030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:22.9320464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:22.9321140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:22.9594106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:22.9594893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:22.9597921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:22.9598533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:23.0197800Z dist init r=2, world=4 2022-09-27T16:15:23.0469969Z dist init r=3, world=4 2022-09-27T16:15:23.0591729Z dist init r=1, world=4 2022-09-27T16:15:23.0867859Z dist init r=0, world=4 2022-09-27T16:15:23.1009190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:23.1205203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:23.1205788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:23.1206159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:23.1206752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:23.1207275Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:23.1207970Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:23.1212307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:23.1312428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:23.1312934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:23.1313473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:23.1313992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:23.4834908Z skip: Need at least 2 CUDA devices (1.720s) 2022-09-27T16:15:23.4905308Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10460 2022-09-27T16:15:23.4940906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10461 2022-09-27T16:15:23.4970223Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10462 2022-09-27T16:15:23.4999524Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10463 2022-09-27T16:15:24.6000415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:24.6001185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:24.6004316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:24.6005023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:24.6251670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:24.6252361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:24.6256133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:24.6257074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:24.6418823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:24.6419469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:24.6422605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:24.6423215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:24.6744002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:24.6744618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:24.6747698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:24.6748339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:24.7289038Z dist init r=2, world=4 2022-09-27T16:15:24.7538312Z dist init r=3, world=4 2022-09-27T16:15:24.7698306Z dist init r=1, world=4 2022-09-27T16:15:24.8019013Z dist init r=0, world=4 2022-09-27T16:15:24.8251423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:24.8453571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:24.8556083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:24.8556517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:24.8557077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:24.8557572Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:24.8558153Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:24.8656203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:24.8664071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:24.8664629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:24.8665163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:24.8665721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:25.2040474Z skip: Need at least 2 CUDA devices (1.720s) 2022-09-27T16:15:25.2109515Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10516 2022-09-27T16:15:25.2143775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10517 2022-09-27T16:15:25.2171542Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10518 2022-09-27T16:15:25.2198717Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10519 2022-09-27T16:15:26.4342423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:26.4342945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:26.4346765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:26.4347277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:26.4695988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:26.4696628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:26.4699932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:26.4700558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:26.4818205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:26.4822428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:26.4823232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:26.4823748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:26.4855111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:26.4855715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:26.4859867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:26.4860495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:26.5627438Z dist init r=2, world=4 2022-09-27T16:15:26.5970222Z dist init r=0, world=4 2022-09-27T16:15:26.6086590Z dist init r=1, world=4 2022-09-27T16:15:26.6125886Z dist init r=3, world=4 2022-09-27T16:15:26.6398055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:26.6500087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:26.6500716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:26.6501261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:26.6501879Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:26.6502454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:26.6503029Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:26.6503553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:26.6609935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:26.6610514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:26.6611212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:26.6611814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:27.0241554Z skip: Need at least 2 CUDA devices (1.820s) 2022-09-27T16:15:27.0311886Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10572 2022-09-27T16:15:27.0346102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10573 2022-09-27T16:15:27.0374748Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10574 2022-09-27T16:15:27.0403301Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10575 2022-09-27T16:15:28.1481036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:28.1481699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:28.1485085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:28.1485720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:28.2029114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:28.2029752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:28.2033403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:28.2034064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:28.2038901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:28.2039511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:28.2043386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:28.2046484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:28.2108980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:28.2109564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:28.2113488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:28.2114095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:28.2764127Z dist init r=3, world=4 2022-09-27T16:15:28.3305993Z dist init r=0, world=4 2022-09-27T16:15:28.3312896Z dist init r=2, world=4 2022-09-27T16:15:28.3390761Z dist init r=1, world=4 2022-09-27T16:15:28.3575485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:28.3702343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:28.3702855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:28.3703284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:28.3703893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:28.3704478Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:28.3705020Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:28.3778823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:28.3812006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:28.3812584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:28.3813088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:28.3813615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:28.7445460Z skip: Need at least 2 CUDA devices (1.720s) 2022-09-27T16:15:28.7516283Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10628 2022-09-27T16:15:28.7550957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10629 2022-09-27T16:15:28.7580009Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10630 2022-09-27T16:15:28.7609438Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10631 2022-09-27T16:15:29.8691937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:29.8692592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:29.8696467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:29.8697094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:29.8831364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:29.8831952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:29.8836069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:29.8940738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:29.8941661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:29.8942271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:29.8946630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:29.8947253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:29.9184028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:29.9184683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:29.9187511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:29.9188159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:29.9982425Z dist init r=3, world=4 2022-09-27T16:15:30.0116790Z dist init r=2, world=4 2022-09-27T16:15:30.0220654Z dist init r=0, world=4 2022-09-27T16:15:30.0468161Z dist init r=1, world=4 2022-09-27T16:15:30.0730020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:30.0831501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:30.0832087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:30.0833346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:30.0833835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:30.0834763Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:30.0835542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:30.0836109Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:30.0840826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:30.0841384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:30.0841920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:30.0842450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:30.3649493Z skip: Need at least 2 CUDA devices (1.620s) 2022-09-27T16:15:30.3717460Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10684 2022-09-27T16:15:30.3749534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10685 2022-09-27T16:15:30.3777404Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10686 2022-09-27T16:15:30.3805804Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10687 2022-09-27T16:15:31.5174216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:31.5174878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:31.5178706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:31.5179343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:31.5332421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:31.5333040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:31.5335999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:31.5336610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:31.5458964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:31.5459880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:31.5463283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:31.5464080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:31.5466022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:31.5466618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:31.5471460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:31.5472042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:31.6469390Z dist init r=3, world=4 2022-09-27T16:15:31.6615119Z dist init r=2, world=4 2022-09-27T16:15:31.6727262Z dist init r=0, world=4 2022-09-27T16:15:31.6749063Z dist init r=1, world=4 2022-09-27T16:15:31.7139890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:31.7140387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:31.7240974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:31.7242904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:31.7243683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:31.7244247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:31.7244769Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:31.7245291Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:31.7248693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:31.7249186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:31.7251726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:31.7252321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:32.0847904Z skip: Need at least 2 CUDA devices (1.720s) 2022-09-27T16:15:32.0918703Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10740 2022-09-27T16:15:32.0954083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10741 2022-09-27T16:15:32.0982931Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10742 2022-09-27T16:15:32.1012278Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10743 2022-09-27T16:15:33.2498577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:33.2499202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:33.2502538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:33.2503182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:33.2579239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:33.2579812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:33.2583942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:33.2584560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:33.2620591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:33.2621179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:33.2626522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:33.2627110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:33.2638559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:33.2639175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:33.2643213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:33.2643855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:33.3855918Z dist init r=0, world=4 2022-09-27T16:15:33.3909912Z dist init r=1, world=4 2022-09-27T16:15:33.3910844Z dist init r=2, world=4 2022-09-27T16:15:33.3954153Z dist init r=3, world=4 2022-09-27T16:15:33.4221618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:33.4323308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:33.4425492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:33.4426436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:33.4427692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:33.4428605Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:33.4429424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:33.4430291Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:33.4433043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:33.4433562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:33.4434095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:33.4434800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:33.8053050Z skip: Need at least 2 CUDA devices (1.720s) 2022-09-27T16:15:33.8124647Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10796 2022-09-27T16:15:33.8160565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10797 2022-09-27T16:15:33.8189906Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10798 2022-09-27T16:15:33.8220014Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10799 2022-09-27T16:15:34.9562579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:34.9563248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:34.9567361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:34.9568139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:34.9605759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:34.9606532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:34.9610892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:34.9611421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:34.9917597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:34.9918253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:34.9922743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:34.9923343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:34.9932612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:34.9933189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:34.9937889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:34.9938680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:35.0841936Z dist init r=3, world=4 2022-09-27T16:15:35.0880872Z dist init r=0, world=4 2022-09-27T16:15:35.1213117Z dist init r=2, world=4 2022-09-27T16:15:35.1213455Z dist init r=1, world=4 2022-09-27T16:15:35.1422316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:35.1522917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:35.1523535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:35.1524107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:35.1524906Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:35.1525428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:35.1525959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:35.1529495Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:35.1630134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:35.1630766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:35.1631308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:35.1631860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:35.5261534Z skip: Need at least 2 CUDA devices (1.721s) 2022-09-27T16:15:35.5333486Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10852 2022-09-27T16:15:35.5367989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10853 2022-09-27T16:15:35.5396986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10854 2022-09-27T16:15:35.5425801Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10855 2022-09-27T16:15:36.6708003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:36.6708685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:36.6711113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:36.6711730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:36.6729918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:36.6730528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:36.6734291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:36.6734918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:36.6866629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:36.6867280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:36.6871318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:36.6871971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:36.7110034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:36.7110756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:36.7114228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:36.7115117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:36.8005632Z dist init r=1, world=4 2022-09-27T16:15:36.8015581Z dist init r=3, world=4 2022-09-27T16:15:36.8152808Z dist init r=0, world=4 2022-09-27T16:15:36.8378039Z dist init r=2, world=4 2022-09-27T16:15:36.8525882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:36.8627464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:36.8730325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:36.8731274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:36.8731750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:36.8732455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:36.8733004Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:36.8733553Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:36.8736485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:36.8737062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:36.8740520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:36.8741126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:37.2466931Z skip: Need at least 2 CUDA devices (1.720s) 2022-09-27T16:15:37.2536956Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10908 2022-09-27T16:15:37.2571246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10909 2022-09-27T16:15:37.2600995Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10910 2022-09-27T16:15:37.2629281Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10911 2022-09-27T16:15:38.4524824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:38.4525481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:38.4527415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:38.4527959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:38.4662594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:38.4663235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:38.4667014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:38.4667615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:38.5150165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:38.5150933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:38.5153506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:38.5154164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:38.5714809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:38.5715470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:38.5718585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:38.5719236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:38.5818918Z dist init r=1, world=4 2022-09-27T16:15:38.6136521Z dist init r=0, world=4 2022-09-27T16:15:38.6452093Z dist init r=2, world=4 2022-09-27T16:15:38.7010594Z dist init r=3, world=4 2022-09-27T16:15:38.7152291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:38.7152703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:38.7255549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:38.7256570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:38.7257157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:38.7257931Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:38.7258777Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:38.7259553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:38.7261293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:38.7261761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:38.7267314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:38.7279521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:39.0671172Z skip: Need at least 2 CUDA devices (1.820s) 2022-09-27T16:15:39.0740767Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10964 2022-09-27T16:15:39.0774266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10965 2022-09-27T16:15:39.0802658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10966 2022-09-27T16:15:39.0831447Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10967 2022-09-27T16:15:40.2072184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:40.2072828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:40.2076565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:40.2077215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:40.2354940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:40.2355454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:40.2360043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:40.2360661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:40.2378226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:40.2378849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:40.2382906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:40.2383421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:40.2743632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:40.2744327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:40.2748188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:40.2748805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:40.3357440Z dist init r=2, world=4 2022-09-27T16:15:40.3651800Z dist init r=1, world=4 2022-09-27T16:15:40.3652194Z dist init r=0, world=4 2022-09-27T16:15:40.4014081Z dist init r=3, world=4 2022-09-27T16:15:40.4163304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:40.4263946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:40.4365655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:40.4366967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:40.4367768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:40.4368864Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:40.4369693Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:40.4370279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:40.4374552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:40.4376048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:40.4376709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:40.4377398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:40.7873315Z skip: Need at least 2 CUDA devices (1.720s) 2022-09-27T16:15:40.7943167Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11020 2022-09-27T16:15:40.7977474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11021 2022-09-27T16:15:40.8024772Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11022 2022-09-27T16:15:40.8040089Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11023 2022-09-27T16:15:41.8995185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:41.8995795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:41.8998816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:41.8999378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:41.9567076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:41.9567674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:41.9570230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:41.9570779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:41.9576656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:41.9577387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:41.9581330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:41.9582030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:41.9801997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:41.9802633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:41.9805162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:41.9805804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:42.0292425Z dist init r=3, world=4 2022-09-27T16:15:42.0847528Z dist init r=2, world=4 2022-09-27T16:15:42.0867358Z dist init r=0, world=4 2022-09-27T16:15:42.1068987Z dist init r=1, world=4 2022-09-27T16:15:42.1257645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:42.1358133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:42.1407425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:42.1407878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:42.1408523Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:42.1409076Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:42.1461612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:42.1462256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:42.1515020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:42.1515676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:42.1516298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:42.1516929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:42.5082509Z skip: Need at least 2 CUDA devices (1.721s) 2022-09-27T16:15:42.5138291Z test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11076 2022-09-27T16:15:42.5173017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11077 2022-09-27T16:15:42.5201900Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11078 2022-09-27T16:15:42.5231052Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11079 2022-09-27T16:15:43.7106969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:43.7107604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:43.7111027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:43.7111679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:43.7150427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:43.7151201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:43.7154749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:43.7155444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:43.7197894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:43.7198583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:43.7202098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:43.7202804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:43.7402872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:43.7403493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:43.7406957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:43.7407591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:43.8444478Z dist init r=2, world=4 2022-09-27T16:15:43.8483429Z dist init r=1, world=4 2022-09-27T16:15:43.8604402Z dist init r=0, world=4 2022-09-27T16:15:43.8702521Z dist init r=3, world=4 2022-09-27T16:15:43.8995702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:43.8996102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:43.9097483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:43.9098011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:43.9099010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:43.9099553Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:43.9100153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:43.9100660Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:43.9204999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:43.9205575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:43.9206011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:43.9206348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:44.2273319Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:15:44.2326741Z test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11132 2022-09-27T16:15:44.2360886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11133 2022-09-27T16:15:44.2389733Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11134 2022-09-27T16:15:44.2418506Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11135 2022-09-27T16:15:45.3721947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:45.3722583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:45.3725998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:45.3726611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:45.3811085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:45.3811760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:45.3812560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:45.3813137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:45.3815434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:45.3816050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:45.3816753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:45.3817335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:45.4215358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:45.4215958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:45.4218647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:45.4219270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:45.5014782Z dist init r=0, world=4 2022-09-27T16:15:45.5088332Z dist init r=2, world=4 2022-09-27T16:15:45.5096062Z dist init r=1, world=4 2022-09-27T16:15:45.5480340Z dist init r=3, world=4 2022-09-27T16:15:45.5587975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:45.5626799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:45.5627675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:45.5628114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:45.5628787Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:45.5629324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:45.5629848Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:45.5690454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:45.5733773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:45.5734347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:45.5734888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:45.5735426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:45.8459065Z skip: Need at least 2 CUDA devices (1.618s) 2022-09-27T16:15:45.8459296Z 2022-09-27T16:15:45.8459730Z ---------------------------------------------------------------------- 2022-09-27T16:15:45.8460130Z Ran 46 tests in 39.834s 2022-09-27T16:15:45.8460316Z 2022-09-27T16:15:45.8460440Z OK (skipped=46) 2022-09-27T16:15:45.8460605Z 2022-09-27T16:15:45.8460729Z Generating XML reports... 2022-09-27T16:15:45.8515513Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20220927161506.xml 2022-09-27T16:15:45.8540316Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20220927161506.xml 2022-09-27T16:15:46.1599184Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2022-09-27 16:15:46.159447] 2022-09-27T16:15:46.1599815Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:15:46.159555] 2022-09-27T16:15:47.4295940Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm 2022-09-27T16:15:47.4308029Z 2022-09-27T16:15:47.4308137Z Running tests... 2022-09-27T16:15:47.4308928Z ---------------------------------------------------------------------- 2022-09-27T16:15:47.4316981Z test_fsdp_calc_grad_norm_norm_type_1_3_nested_fsdp_False (__main__.TestCalcuGradNorm) 2022-09-27T16:15:47.6782624Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11199 2022-09-27T16:15:47.6810369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11200 2022-09-27T16:15:47.6838539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11201 2022-09-27T16:15:47.6869533Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11202 2022-09-27T16:15:48.9224952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:48.9225605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:48.9229207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:48.9229838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:48.9259243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:48.9259839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:48.9264542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:48.9265428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:48.9393676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:48.9394734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:48.9398251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:48.9398988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:48.9431083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:48.9431717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:48.9434951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:48.9435575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:49.0528543Z dist init r=0, world=4 2022-09-27T16:15:49.0581723Z dist init r=3, world=4 2022-09-27T16:15:49.0709039Z dist init r=1, world=4 2022-09-27T16:15:49.0772072Z dist init r=2, world=4 2022-09-27T16:15:49.0890379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:49.1082540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:49.1184145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:49.1184821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:49.1185664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:49.1186212Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:49.1186735Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:49.1194763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:49.1291926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:49.1292599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:49.1293045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:49.1293504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:49.4915499Z skip: Need at least 2 CUDA devices (2.060s) 2022-09-27T16:15:49.4923395Z test_fsdp_calc_grad_norm_norm_type_1_3_nested_fsdp_True (__main__.TestCalcuGradNorm) 2022-09-27T16:15:49.4968244Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11255 2022-09-27T16:15:49.5002800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11256 2022-09-27T16:15:49.5031194Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11257 2022-09-27T16:15:49.5059713Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11258 2022-09-27T16:15:50.6428751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:50.6429413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:50.6432614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:50.6433196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:50.6551413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:50.6552116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:50.6552996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:50.6553470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:50.6554196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:50.6554969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:50.6556086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:50.6556693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:50.6557470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:50.6558073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:50.6558794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:50.6559364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:50.7710337Z dist init r=3, world=4 2022-09-27T16:15:50.7832012Z dist init r=0, world=4 2022-09-27T16:15:50.7832368Z dist init r=1, world=4 2022-09-27T16:15:50.7835066Z dist init r=2, world=4 2022-09-27T16:15:50.8044599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:50.8146148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:50.8146695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:50.8147349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:50.8148072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:50.8148723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:50.8149262Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:50.8149764Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:50.8254093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:50.8254634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:50.8255174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:50.8255676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:51.2101188Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T16:15:51.2108880Z test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_False (__main__.TestCalcuGradNorm) 2022-09-27T16:15:51.2156904Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11311 2022-09-27T16:15:51.2191875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11312 2022-09-27T16:15:51.2219897Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11313 2022-09-27T16:15:51.2248405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11314 2022-09-27T16:15:52.3331147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:52.3331770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:52.3335451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:52.3336159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:52.3807095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:52.3807617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:52.3810794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:52.3811309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:52.4039086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:52.4039736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:52.4043053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:52.4043689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:52.4165477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:52.4166090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:52.4169472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:52.4170095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:52.4612976Z dist init r=2, world=4 2022-09-27T16:15:52.5075249Z dist init r=1, world=4 2022-09-27T16:15:52.5310279Z dist init r=0, world=4 2022-09-27T16:15:52.5428246Z dist init r=3, world=4 2022-09-27T16:15:52.5626781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:52.5627785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:52.5628808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:52.5629827Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:52.5630551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:52.5631122Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:52.5631642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:52.5728652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:52.5736141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:52.5736808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:52.5737458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:52.5738084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:52.9291250Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:15:52.9297745Z test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_True (__main__.TestCalcuGradNorm) 2022-09-27T16:15:52.9343515Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11367 2022-09-27T16:15:52.9377283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11368 2022-09-27T16:15:52.9405883Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11369 2022-09-27T16:15:52.9434797Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11370 2022-09-27T16:15:54.0982139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:54.0983068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:54.0986085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:54.0986710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:54.1129535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:54.1130163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:54.1134864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:54.1135503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:54.1147707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:54.1148331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:54.1152582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:54.1153191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:54.1166083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:54.1166646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:54.1171387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:54.1171969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:54.2257091Z dist init r=3, world=4 2022-09-27T16:15:54.2448485Z dist init r=2, world=4 2022-09-27T16:15:54.2464325Z dist init r=1, world=4 2022-09-27T16:15:54.2594318Z dist init r=0, world=4 2022-09-27T16:15:54.3007389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:54.3108181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:54.3209936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:54.3210629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:54.3211549Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:54.3212292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:54.3212940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:54.3213527Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:54.3218266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:54.3218791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:54.3219332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:54.3219853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:54.6476976Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T16:15:54.6484484Z test_fsdp_calc_grad_norm_norm_type_2_5_nested_fsdp_False (__main__.TestCalcuGradNorm) 2022-09-27T16:15:54.6532533Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11423 2022-09-27T16:15:54.6566367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11424 2022-09-27T16:15:54.6596086Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11425 2022-09-27T16:15:54.6625543Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11426 2022-09-27T16:15:55.8018428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:55.8018954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:55.8022733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:55.8023358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:55.8284682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:55.8285307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:55.8288748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:55.8289411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:55.8351169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:55.8351777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:55.8355824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:55.8356464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:55.8414562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:55.8415185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:55.8418885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:55.8419515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:55.9294775Z dist init r=2, world=4 2022-09-27T16:15:55.9561035Z dist init r=0, world=4 2022-09-27T16:15:55.9639947Z dist init r=1, world=4 2022-09-27T16:15:55.9686463Z dist init r=3, world=4 2022-09-27T16:15:55.9805181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:55.9896055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:55.9998302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:55.9999031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:55.9999576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:56.0000159Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:56.0000686Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:56.0008893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:56.0106679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:56.0107063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:56.0107522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:56.0108233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:56.3667489Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:15:56.3675076Z test_fsdp_calc_grad_norm_norm_type_2_5_nested_fsdp_True (__main__.TestCalcuGradNorm) 2022-09-27T16:15:56.3720739Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11479 2022-09-27T16:15:56.3755795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11480 2022-09-27T16:15:56.3784293Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11481 2022-09-27T16:15:56.3812433Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11482 2022-09-27T16:15:57.4769408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:57.4769920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:57.4773545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:57.4774180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:57.5255084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:57.5255704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:57.5258983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:57.5259588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:57.5307072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:57.5307673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:57.5312436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:57.5313053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:57.5535295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:57.5535900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:57.5536825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:57.5537409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:57.6057623Z dist init r=2, world=4 2022-09-27T16:15:57.6532057Z dist init r=1, world=4 2022-09-27T16:15:57.6605860Z dist init r=0, world=4 2022-09-27T16:15:57.6807277Z dist init r=3, world=4 2022-09-27T16:15:57.6942357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:57.7044070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:57.7146081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:57.7147164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:57.7147724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:57.7148236Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:57.7148746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:57.7149289Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:57.7253848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:57.7254447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:57.7255062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:57.7255420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:58.0854443Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:15:58.0861779Z test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_False (__main__.TestCalcuGradNorm) 2022-09-27T16:15:58.0909160Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11535 2022-09-27T16:15:58.0943019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11536 2022-09-27T16:15:58.0971704Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11537 2022-09-27T16:15:58.1001543Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11538 2022-09-27T16:15:59.2442782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:59.2443440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:59.2446929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:59.2447556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:59.2719298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:59.2719904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:59.2721584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:59.2722159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:59.2887504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:59.2888120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:59.2891154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:59.2891770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:59.3419921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:15:59.3420574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:15:59.3423476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:15:59.3424098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:15:59.3792825Z dist init r=0, world=4 2022-09-27T16:15:59.4016197Z dist init r=3, world=4 2022-09-27T16:15:59.4161387Z dist init r=1, world=4 2022-09-27T16:15:59.4707919Z dist init r=2, world=4 2022-09-27T16:15:59.4918104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:15:59.5009897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:15:59.5111984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:15:59.5112510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:15:59.5113164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:59.5113711Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:59.5114446Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:59.5120764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:15:59.5219446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:15:59.5220142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:15:59.5220660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:15:59.5221226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:15:59.8042334Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:15:59.8049770Z test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_True (__main__.TestCalcuGradNorm) 2022-09-27T16:15:59.8094276Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11591 2022-09-27T16:15:59.8128144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11592 2022-09-27T16:15:59.8157836Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11593 2022-09-27T16:15:59.8187380Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11594 2022-09-27T16:16:00.9613396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:00.9614070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:00.9617483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:00.9618129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:00.9704573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:00.9708798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:00.9709610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:00.9710242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:00.9888557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:00.9889198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:00.9892448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:00.9893050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:00.9910986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:00.9911536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:00.9915369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:00.9916002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:01.0905953Z dist init r=1, world=4 2022-09-27T16:16:01.0981673Z dist init r=3, world=4 2022-09-27T16:16:01.1164422Z dist init r=2, world=4 2022-09-27T16:16:01.1183330Z dist init r=0, world=4 2022-09-27T16:16:01.1494830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:01.1495415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:01.1595851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:16:01.1596430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:16:01.1597376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:01.1598130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:01.1598734Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:01.1599396Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:01.1704225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:01.1704660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:01.1705171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:16:01.1705697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:16:01.5228936Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:16:01.5236684Z test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm) 2022-09-27T16:16:01.5281672Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11647 2022-09-27T16:16:01.5316920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11648 2022-09-27T16:16:01.5345157Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11649 2022-09-27T16:16:01.5372580Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11650 2022-09-27T16:16:02.6593199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:02.6593722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:02.6597498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:02.6598029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:02.6769244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:02.6769737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:02.6773448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:02.6774435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:02.7097161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:02.7097795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:02.7101950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:02.7102585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:02.7116096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:02.7116737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:02.7121319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:02.7121930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:02.7878044Z dist init r=1, world=4 2022-09-27T16:16:02.8055888Z dist init r=3, world=4 2022-09-27T16:16:02.8382787Z dist init r=2, world=4 2022-09-27T16:16:02.8403362Z dist init r=0, world=4 2022-09-27T16:16:02.8793091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:02.8793953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:16:02.8794353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:02.8795049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:16:02.8795834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:02.8796564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:02.8797090Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:02.8797587Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:02.8900669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:02.8901348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:16:02.8901968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:16:02.8902478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:03.2415194Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T16:16:03.2421049Z test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm) 2022-09-27T16:16:03.2466858Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11703 2022-09-27T16:16:03.2500681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11704 2022-09-27T16:16:03.2528704Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11705 2022-09-27T16:16:03.2558076Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11706 2022-09-27T16:16:04.4440984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:04.4441657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:04.4444601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:04.4445258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:04.4545022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:04.4545638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:04.4549385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:04.4550023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:04.4619068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:04.4619783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:04.4623721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:04.4624363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:04.4778083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:04.4778714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:04.4783142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:04.4783785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:04.5720927Z dist init r=3, world=4 2022-09-27T16:16:04.5904400Z dist init r=2, world=4 2022-09-27T16:16:04.5969735Z dist init r=1, world=4 2022-09-27T16:16:04.6054951Z dist init r=0, world=4 2022-09-27T16:16:04.6359070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:04.6460212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:04.6460786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:16:04.6461273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:16:04.6462267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:04.6462799Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:04.6463353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:04.6463870Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:04.6569094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:16:04.6569681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:04.6570210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:04.6570732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:16:04.9599808Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T16:16:04.9605721Z test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm) 2022-09-27T16:16:04.9653046Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11759 2022-09-27T16:16:04.9686958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11760 2022-09-27T16:16:04.9714482Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11761 2022-09-27T16:16:04.9742913Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11762 2022-09-27T16:16:06.1660010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:06.1660540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:06.1661147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:06.1663587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:06.1664453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:06.1665040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:06.1665799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:06.1666355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:06.1705166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:06.1705682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:06.1710116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:06.1710767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:06.2100242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:06.2101160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:06.2103091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:06.2103911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:06.2961965Z dist init r=2, world=4 2022-09-27T16:16:06.2962348Z dist init r=3, world=4 2022-09-27T16:16:06.2995936Z dist init r=1, world=4 2022-09-27T16:16:06.3381547Z dist init r=0, world=4 2022-09-27T16:16:06.3573809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:06.3777686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:16:06.3778387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:06.3779433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:06.3780157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:16:06.3780985Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:06.3781804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:06.3782603Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:06.3784433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:06.3785065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:16:06.3785863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:16:06.3786333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:06.6784962Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T16:16:06.6790441Z test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm) 2022-09-27T16:16:06.6832232Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11815 2022-09-27T16:16:06.6863263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11816 2022-09-27T16:16:06.6891513Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11817 2022-09-27T16:16:06.6927328Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11818 2022-09-27T16:16:07.8142417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:07.8143024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:07.8146163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:07.8416004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:07.8417017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:07.8417545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:07.8420349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:07.8420899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:07.8426075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:07.8426663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:07.8431292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:07.8431921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:07.8761027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:07.8761690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:07.8764883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:07.8765508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:07.9434433Z dist init r=0, world=4 2022-09-27T16:16:07.9695750Z dist init r=1, world=4 2022-09-27T16:16:07.9703929Z dist init r=3, world=4 2022-09-27T16:16:08.0031386Z dist init r=2, world=4 2022-09-27T16:16:08.0308775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:08.0309174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:08.0409639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:16:08.0410373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:16:08.0411231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:08.0411904Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:08.0412433Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:08.0412939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:08.0517810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:08.0518379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:16:08.0518941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:08.0519389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:16:08.3969417Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T16:16:08.3974477Z test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm) 2022-09-27T16:16:08.4020573Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11871 2022-09-27T16:16:08.4055066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11872 2022-09-27T16:16:08.4085020Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11873 2022-09-27T16:16:08.4114150Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11874 2022-09-27T16:16:09.5481267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:09.5481771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:09.5485004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:09.5485524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:09.5633555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:09.5634172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:09.5638197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:09.5639122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:09.5815890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:09.5816824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:09.5819163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:09.5819780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:09.5914159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:09.5914952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:09.5918768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:09.5919432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:09.6772629Z dist init r=2, world=4 2022-09-27T16:16:09.6916309Z dist init r=0, world=4 2022-09-27T16:16:09.7091715Z dist init r=3, world=4 2022-09-27T16:16:09.7203150Z dist init r=1, world=4 2022-09-27T16:16:09.7401334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:09.7501492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:09.7604118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:16:09.7604806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:16:09.7605857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:09.7606852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:09.7607804Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:09.7608769Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:09.7720983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:16:09.7721664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:09.7722308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:16:09.7738538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:10.1155573Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:16:10.1162060Z test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm) 2022-09-27T16:16:10.1208131Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11927 2022-09-27T16:16:10.1243176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11928 2022-09-27T16:16:10.1273313Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11929 2022-09-27T16:16:10.1303946Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11930 2022-09-27T16:16:11.2951323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:11.2951954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:11.2955127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:11.2955759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:11.2960592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:11.2961375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:11.2964211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:11.2964916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:11.3206958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:11.3207589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:11.3211226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:11.3211958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:11.3428735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:11.3432396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:11.3433259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:11.3433869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:11.4256927Z dist init r=1, world=4 2022-09-27T16:16:11.4261279Z dist init r=0, world=4 2022-09-27T16:16:11.4480350Z dist init r=3, world=4 2022-09-27T16:16:11.4716852Z dist init r=2, world=4 2022-09-27T16:16:11.4890463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:11.4991102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:11.5071723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:16:11.5072184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:16:11.5072908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:11.5073448Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:11.5093866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:11.5094515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:11.5179028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:11.5179653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:11.5180184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:16:11.5180537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:16:11.8345900Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:16:11.8351977Z test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm) 2022-09-27T16:16:11.8394827Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11983 2022-09-27T16:16:11.8427662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11984 2022-09-27T16:16:11.8458898Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11985 2022-09-27T16:16:11.8496776Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11986 2022-09-27T16:16:12.9537350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:12.9538156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:12.9540574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:12.9541236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:12.9844081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:12.9844613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:12.9847422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:12.9847925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:13.0005919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:13.0006513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:13.0010429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:13.0011063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:13.0178342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:13.0178969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:13.0182405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:13.0183049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:13.0811612Z dist init r=0, world=4 2022-09-27T16:16:13.1110297Z dist init r=3, world=4 2022-09-27T16:16:13.1263934Z dist init r=1, world=4 2022-09-27T16:16:13.1452549Z dist init r=2, world=4 2022-09-27T16:16:13.1621141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:13.1823846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:16:13.1824520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:13.1825553Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:13.1826260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:16:13.1827170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:13.1828024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:13.1828557Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:13.1931523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:16:13.1932213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:13.1932848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:16:13.1935492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:13.5538132Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:16:13.5544601Z test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm) 2022-09-27T16:16:13.5594226Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12039 2022-09-27T16:16:13.5631881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12040 2022-09-27T16:16:13.5664475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12041 2022-09-27T16:16:13.5695158Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12042 2022-09-27T16:16:14.7040577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:14.7041225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:14.7045266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:14.7045889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:14.7299756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:14.7300364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:14.7303322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:14.7303957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:14.7386593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:14.7387183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:14.7390072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:14.7390731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:14.7391491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:14.7392119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:14.7395965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:14.7396568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:14.8315859Z dist init r=1, world=4 2022-09-27T16:16:14.8566598Z dist init r=2, world=4 2022-09-27T16:16:14.8656285Z dist init r=0, world=4 2022-09-27T16:16:14.8666827Z dist init r=3, world=4 2022-09-27T16:16:14.8877024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:14.9080117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:14.9080715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:16:14.9081340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:16:14.9082170Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:14.9082863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:14.9083548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:14.9084071Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:16:14.9088116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:16:14.9088580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:16:14.9090232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:14.9090790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:15.2737202Z skip: Need at least 2 CUDA devices (1.720s) 2022-09-27T16:16:15.2737522Z 2022-09-27T16:16:15.2738286Z ---------------------------------------------------------------------- 2022-09-27T16:16:15.2738585Z Ran 16 tests in 27.843s 2022-09-27T16:16:15.2738853Z 2022-09-27T16:16:15.2738932Z OK (skipped=16) 2022-09-27T16:16:15.2739047Z 2022-09-27T16:16:15.2739133Z Generating XML reports... 2022-09-27T16:16:15.2779242Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220927161547.xml 2022-09-27T16:16:15.2789085Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220927161547.xml 2022-09-27T16:16:15.5855083Z Running distributed/fsdp/test_fsdp_misc ... [2022-09-27 16:16:15.585102] 2022-09-27T16:16:15.5855635Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:16:15.585188] 2022-09-27T16:16:16.8219694Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_misc 2022-09-27T16:16:16.8231311Z 2022-09-27T16:16:16.8231651Z Running tests... 2022-09-27T16:16:16.8232075Z ---------------------------------------------------------------------- 2022-09-27T16:16:16.8239950Z test_cpu_init_with_sync_module_states (__main__.TestFSDPMisc) 2022-09-27T16:16:17.0746530Z Tests that passing ``sync_module_states=True`` raises an error for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12106 2022-09-27T16:16:17.0774243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12107 2022-09-27T16:16:18.1625964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:18.1626352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:18.1628992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:18.1629381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:18.1638963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:18.1639377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:18.1643547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:18.1644021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:18.2883216Z dist init r=0, world=2 2022-09-27T16:16:18.2886664Z dist init r=1, world=2 2022-09-27T16:16:18.2995220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:18.2995823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:18.2996459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:18.2996991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:18.3101264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:18.3101875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:18.5807633Z skip: Need at least 2 CUDA devices (1.757s) 2022-09-27T16:16:18.5815624Z test_device_id_auto_wrap (__main__.TestFSDPMisc) 2022-09-27T16:16:18.5860284Z Tests that ``auto_wrap_policy`` propagates ``device_id`` to all ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12134 2022-09-27T16:16:18.5893967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12135 2022-09-27T16:16:19.6814614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:19.6815195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:19.6818136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:19.6818647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:19.6825754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:19.6826158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:19.6830207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:19.6830613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:19.8073580Z dist init r=0, world=2 2022-09-27T16:16:19.8092844Z dist init r=1, world=2 2022-09-27T16:16:19.8282894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:19.8283433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:19.8284165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:19.8284686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:19.8288778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:19.8289367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:20.0924124Z skip: Need at least 2 CUDA devices (1.512s) 2022-09-27T16:16:20.0934011Z test_fsdp_cpu_init_stays_on_cpu (__main__.TestFSDPMisc) 2022-09-27T16:16:20.0978380Z Tests that passing a CPU module to FSDP preserves that the wrapped ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12162 2022-09-27T16:16:20.1010542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12163 2022-09-27T16:16:21.1881775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:21.1882233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:21.1885065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:21.1885605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:21.1896921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:21.1897260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:21.1901543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:21.1901901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:21.3146516Z dist init r=1, world=2 2022-09-27T16:16:21.3164910Z dist init r=0, world=2 2022-09-27T16:16:21.3374568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:21.3375240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:21.3375893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:21.3376422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:21.3380001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:21.3380402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:21.6042109Z skip: Need at least 2 CUDA devices (1.512s) 2022-09-27T16:16:21.6051538Z test_fsdp_device_id_cpu_offload (__main__.TestFSDPMisc) 2022-09-27T16:16:21.6097325Z Ensures that even if device_id is specified but we have ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12190 2022-09-27T16:16:21.6131791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12191 2022-09-27T16:16:22.7061051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:22.7061707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:22.7064728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:22.7065346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:22.7113931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:22.7114390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:22.7118256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:22.7118753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:22.8308897Z dist init r=0, world=2 2022-09-27T16:16:22.8362186Z dist init r=1, world=2 2022-09-27T16:16:22.8517469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:22.8517899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:22.8518629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:22.8519153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:22.8624008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:22.8624590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:23.1162468Z skip: Need at least 2 CUDA devices (1.512s) 2022-09-27T16:16:23.1178203Z test_fsdp_device_id_use_index_False (__main__.TestFSDPMisc) 2022-09-27T16:16:23.1223164Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12218 2022-09-27T16:16:23.1257970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12219 2022-09-27T16:16:24.2254286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:24.2254926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:24.2258452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:24.2259057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:24.2327081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:24.2330843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:24.2331659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:24.2332294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:24.3510752Z dist init r=0, world=2 2022-09-27T16:16:24.3567946Z dist init r=1, world=2 2022-09-27T16:16:24.3676189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:24.3676785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:24.3677753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:24.3678291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:24.3782794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:24.3783193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:24.6290948Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:16:24.6306469Z test_fsdp_device_id_use_index_True (__main__.TestFSDPMisc) 2022-09-27T16:16:24.6351492Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12246 2022-09-27T16:16:24.6387045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12247 2022-09-27T16:16:25.7394861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:25.7395713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:25.7425947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:25.7426606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:25.7586518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:25.7587104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:25.7589885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:25.7590492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:25.8667838Z dist init r=0, world=2 2022-09-27T16:16:25.8842986Z dist init r=1, world=2 2022-09-27T16:16:25.8951390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:25.8952030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:25.8952664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:25.8953193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:25.9057324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:25.9057695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:26.2420862Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:16:26.2480664Z test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12274 2022-09-27T16:16:26.2515513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12275 2022-09-27T16:16:27.3464932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:27.3465297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:27.3468019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:27.3468433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:27.3476046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:27.3476419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:27.3480931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:27.3481612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:27.4745680Z dist init r=0, world=2 2022-09-27T16:16:27.4755739Z dist init r=1, world=2 2022-09-27T16:16:27.4954691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:27.4955161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:27.4955977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:27.4956513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:27.4960771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:27.4961328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:27.7549597Z skip: Need at least 2 CUDA devices (1.513s) 2022-09-27T16:16:27.7608684Z test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12302 2022-09-27T16:16:27.7642731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12303 2022-09-27T16:16:28.8676202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:28.8676871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:28.8680781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:28.8681414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:28.8707375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:28.8707996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:28.8711846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:28.8712452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:28.9936279Z dist init r=1, world=2 2022-09-27T16:16:28.9963288Z dist init r=0, world=2 2022-09-27T16:16:29.0172352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:29.0173366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:29.0174067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:29.0174607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:29.0178036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:29.0178594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:29.2673975Z skip: Need at least 2 CUDA devices (1.512s) 2022-09-27T16:16:29.2729496Z test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12330 2022-09-27T16:16:29.2758732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12331 2022-09-27T16:16:30.3688851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:30.3689221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:30.3692180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:30.3692983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:30.3693493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:30.3693914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:30.3698730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:30.3699179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:30.4947186Z dist init r=1, world=2 2022-09-27T16:16:30.4953167Z dist init r=0, world=2 2022-09-27T16:16:30.5163895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:30.5164582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:30.5165208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:30.5165723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:30.5268747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:30.5269324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:30.7792385Z skip: Need at least 2 CUDA devices (1.512s) 2022-09-27T16:16:30.7851091Z test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12358 2022-09-27T16:16:30.7883849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12359 2022-09-27T16:16:31.8871435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:31.8872117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:31.8874781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:31.8875372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:31.8919993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:31.8920577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:31.8924974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:31.8925581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:32.0135295Z dist init r=0, world=2 2022-09-27T16:16:32.0187263Z dist init r=1, world=2 2022-09-27T16:16:32.0295455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:32.0295890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:32.0296666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:32.0297200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:32.0402266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:32.0402802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:32.2917492Z skip: Need at least 2 CUDA devices (1.512s) 2022-09-27T16:16:32.2970875Z test_fsdp_namedtuple (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12386 2022-09-27T16:16:32.3004220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12387 2022-09-27T16:16:33.3988562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:33.3989161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:33.3991465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:33.3992000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:33.3992769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:33.3993238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:33.3997768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:33.3998155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:33.5250448Z dist init r=0, world=2 2022-09-27T16:16:33.5250666Z dist init r=1, world=2 2022-09-27T16:16:33.5458848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:33.5459563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:33.5460161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:33.5460678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:33.5564725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:33.5565322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:33.8039109Z skip: Need at least 2 CUDA devices (1.512s) 2022-09-27T16:16:33.8108506Z test_fsdp_not_all_outputs_used_in_loss (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12414 2022-09-27T16:16:33.8140873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12415 2022-09-27T16:16:34.9207493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:34.9208157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:34.9208861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:34.9209367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:34.9211308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:34.9211852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:34.9212645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:34.9213200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:35.0475376Z dist init r=1, world=2 2022-09-27T16:16:35.0476070Z dist init r=0, world=2 2022-09-27T16:16:35.0686773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:35.0687525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:35.0688373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:35.0689176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:35.0691962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:35.0692713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:35.4176205Z skip: Need at least 2 CUDA devices (1.614s) 2022-09-27T16:16:35.4188321Z test_fsdp_same_model_across_ranks (__main__.TestFSDPMisc) 2022-09-27T16:16:35.4232708Z FSDP broadcasts model from rank 0 to ensure it starts off with the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12442 2022-09-27T16:16:35.4265677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12443 2022-09-27T16:16:36.5419823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:36.5420210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:36.5423225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:36.5423578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:36.5521972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:36.5522323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:36.5526744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:36.5527135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:36.6686774Z dist init r=1, world=2 2022-09-27T16:16:36.6789582Z dist init r=0, world=2 2022-09-27T16:16:36.6997916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:36.6998582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:36.6999422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:36.7000238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:36.7003680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:36.7004102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:37.0301470Z skip: Need at least 2 CUDA devices (1.612s) 2022-09-27T16:16:37.0307643Z test_module_device_mismatches_device_id (__main__.TestFSDPMisc) 2022-09-27T16:16:37.0354734Z Tests that specifying a ``device_id`` argument to FSDP for a GPU ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12470 2022-09-27T16:16:37.0388228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12471 2022-09-27T16:16:38.1569250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:38.1569616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:38.1572477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:38.1573102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:38.1932988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:38.1933470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:38.1936676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:38.1937185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:38.2845262Z dist init r=0, world=2 2022-09-27T16:16:38.3217104Z dist init r=1, world=2 2022-09-27T16:16:38.3355783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:38.3356662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:38.3357578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:38.3358219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:38.3462805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:38.3463197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:38.6421038Z skip: Need at least 2 CUDA devices (1.612s) 2022-09-27T16:16:38.6427177Z test_multi_device_not_supported (__main__.TestFSDPMisc) 2022-09-27T16:16:38.6472922Z Tests that wrapping a multi-device module (i.e. with submodules on ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12498 2022-09-27T16:16:38.6506599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12499 2022-09-27T16:16:39.7494407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:39.7494787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:39.7498359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:39.7498805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:39.7499565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:39.7500036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:39.7504416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:39.7504914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:39.8757129Z dist init r=1, world=2 2022-09-27T16:16:39.8764851Z dist init r=0, world=2 2022-09-27T16:16:39.8966008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:39.8966593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:39.8967468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:39.8968269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:39.8971544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:39.8972098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:40.1537417Z skip: Need at least 2 CUDA devices (1.512s) 2022-09-27T16:16:40.1545518Z test_no_params (__main__.TestFSDPMisc) 2022-09-27T16:16:40.1590087Z Test that device_id and cpu init work if module has no params ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12526 2022-09-27T16:16:40.1623475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12527 2022-09-27T16:16:41.2734955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:41.2735338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:41.2738403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:41.2738800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:41.3032734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:41.3033236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:41.3037112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:41.3037555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:41.4003927Z dist init r=1, world=2 2022-09-27T16:16:41.4270217Z dist init r=0, world=2 2022-09-27T16:16:41.4413754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:16:41.4414208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:16:41.4414842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:41.4415454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:16:41.4419677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:16:41.4420251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:16:41.7656645Z skip: Need at least 2 CUDA devices (1.612s) 2022-09-27T16:16:41.7656923Z 2022-09-27T16:16:41.7657366Z ---------------------------------------------------------------------- 2022-09-27T16:16:41.7657753Z Ran 16 tests in 24.942s 2022-09-27T16:16:41.7657943Z 2022-09-27T16:16:41.7658054Z OK (skipped=16) 2022-09-27T16:16:41.7658221Z 2022-09-27T16:16:41.7658335Z Generating XML reports... 2022-09-27T16:16:41.7707111Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20220927161616.xml 2022-09-27T16:16:42.0859399Z Running distributed/elastic/multiprocessing/api_test ... [2022-09-27 16:16:42.085551] 2022-09-27T16:16:42.0860005Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:16:42.085655] 2022-09-27T16:16:43.3986413Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test 2022-09-27T16:16:43.4000538Z 2022-09-27T16:16:43.4000666Z Running tests... 2022-09-27T16:16:43.4001096Z ---------------------------------------------------------------------- 2022-09-27T16:16:43.6387624Z test_get_failures (__main__.RunProcResultsTest) ... ok (0.238s) 2022-09-27T16:16:43.6407582Z test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s) 2022-09-27T16:16:43.6422639Z test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.001s) 2022-09-27T16:16:43.6679969Z test_binary (__main__.StartProcessesListTest) ... hello stdout from 0 2022-09-27T16:16:43.6680451Z hello stderr from 0 2022-09-27T16:16:43.6716325Z hello stdout from 1 2022-09-27T16:16:43.6716693Z hello stderr from 1 2022-09-27T16:16:43.7538286Z ok (0.111s) 2022-09-27T16:16:43.7859915Z test_binary_exit (__main__.StartProcessesListTest) ... bar stdout from 1 2022-09-27T16:16:43.7860389Z bar stderr from 1 2022-09-27T16:16:43.8672327Z failed (exitcode: 138) local_rank: 0 (pid: 12566) of binary: distributed/elastic/multiprocessing/bin/echo1.py 2022-09-27T16:16:43.8683004Z ok (0.114s) 2022-09-27T16:16:43.8765604Z test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.008s) 2022-09-27T16:16:43.9021480Z test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last): 2022-09-27T16:16:43.9021920Z File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in 2022-09-27T16:16:43.9022212Z raise RuntimeError(f"raised from {rank}") 2022-09-27T16:16:43.9022463Z RuntimeError: raised from 0 2022-09-27T16:16:43.9054495Z bar from 1 2022-09-27T16:16:43.9869399Z failed (exitcode: 1) local_rank: 0 (pid: 12569) of binary: distributed/elastic/multiprocessing/bin/echo2.py 2022-09-27T16:16:43.9878139Z ok (0.111s) 2022-09-27T16:16:44.0189960Z test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1 2022-09-27T16:16:44.1007314Z [trainer0]:hello stdout from 0 2022-09-27T16:16:44.1007576Z [trainer1]:world stderr from 1 2022-09-27T16:16:45.1032392Z ok (1.115s) 2022-09-27T16:16:46.3433374Z test_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:46.3433813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:46.3436040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:46.3436663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:46.3496831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:46.3497211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:46.3499868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:46.3500212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:46.4661223Z hello stdout from 0 2022-09-27T16:16:46.4661615Z hello stderr from 0 2022-09-27T16:16:46.4711266Z hello stdout from 1 2022-09-27T16:16:46.4711493Z hello stderr from 1 2022-09-27T16:16:46.7569119Z Closing process 12576 via signal SIGTERM 2022-09-27T16:16:46.7587142Z ok (1.655s) 2022-09-27T16:16:47.9849234Z test_function_large_ret_val (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:47.9849981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:47.9850721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:47.9851336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:48.0259136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:48.0259876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:48.0262762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:48.0263464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:48.0274731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:48.0275429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:48.0279216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:48.0279941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:48.0313547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:48.0314125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:48.0316777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:48.0317401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:48.7121784Z Closing process 12596 via signal SIGTERM 2022-09-27T16:16:48.7122189Z Closing process 12597 via signal SIGTERM 2022-09-27T16:16:48.7146753Z ok (1.956s) 2022-09-27T16:16:48.7164572Z test_function_raise (__main__.StartProcessesListTest) 2022-09-27T16:16:49.9653964Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:49.9654424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:49.9656469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:49.9656932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:49.9905092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:49.9905706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:49.9908134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:49.9908717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:50.3706095Z failed (exitcode: 1) local_rank: 0 (pid: 12636) of fn: echo2 (start_method: spawn) 2022-09-27T16:16:50.3706380Z Traceback (most recent call last): 2022-09-27T16:16:50.3706959Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll 2022-09-27T16:16:50.3707306Z self._pc.join(-1) 2022-09-27T16:16:50.3707722Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join 2022-09-27T16:16:50.3708062Z raise ProcessRaisedException(msg, error_index, failed_process.pid) 2022-09-27T16:16:50.3708448Z torch.multiprocessing.spawn.ProcessRaisedException: 2022-09-27T16:16:50.3708648Z 2022-09-27T16:16:50.3708812Z -- Process 0 terminated with the following error: 2022-09-27T16:16:50.3709085Z Traceback (most recent call last): 2022-09-27T16:16:50.3709459Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap 2022-09-27T16:16:50.3709761Z fn(i, *args) 2022-09-27T16:16:50.3710132Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap 2022-09-27T16:16:50.3710423Z ret = record(fn)(*args_) 2022-09-27T16:16:50.3710964Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper 2022-09-27T16:16:50.3711318Z return f(*args, **kwargs) 2022-09-27T16:16:50.3711602Z File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2 2022-09-27T16:16:50.3711883Z raise RuntimeError(msg) 2022-09-27T16:16:50.3712128Z RuntimeError: hello 2022-09-27T16:16:50.3712245Z 2022-09-27T16:16:50.3722261Z ok (1.657s) 2022-09-27T16:16:50.3743920Z test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s) 2022-09-27T16:16:50.3758022Z test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.001s) 2022-09-27T16:16:50.3811505Z test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 12656 via signal SIGTERM 2022-09-27T16:16:50.3835405Z ok (0.008s) 2022-09-27T16:16:50.3870526Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn) 2022-09-27T16:16:50.3870976Z Traceback (most recent call last): 2022-09-27T16:16:50.3871405Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll 2022-09-27T16:16:50.3871714Z self._pc.join(-1) 2022-09-27T16:16:50.3871957Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__ 2022-09-27T16:16:50.3872218Z return _mock_self._mock_call(*args, **kwargs) 2022-09-27T16:16:50.3872492Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call 2022-09-27T16:16:50.3872708Z raise effect 2022-09-27T16:16:50.3872984Z torch.multiprocessing.spawn.ProcessRaisedException: test msg 2022-09-27T16:16:50.3881086Z ok (0.004s) 2022-09-27T16:16:51.5497492Z test_pcontext_wait (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:51.5498008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:51.5499384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:51.5499848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:52.9251288Z ok (2.537s) 2022-09-27T16:16:52.9331939Z test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 12667 closing signal SIGTERM 2022-09-27T16:16:52.9346616Z ok (0.010s) 2022-09-27T16:16:52.9369760Z test_to_map (__main__.StartProcessesListTest) ... ok (0.002s) 2022-09-27T16:16:52.9378100Z test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s) 2022-09-27T16:16:54.1821648Z test_void_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:54.1822086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:54.1823057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:54.1823668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:54.1854402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:54.1854886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:54.1857250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:54.1857602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:54.3065491Z world 2022-09-27T16:16:54.3083969Z hello 2022-09-27T16:16:54.5924022Z Closing process 12669 via signal SIGTERM 2022-09-27T16:16:54.5935151Z ok (1.656s) 2022-09-27T16:16:54.5957217Z test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s) 2022-09-27T16:16:54.6279447Z test_binary_exit (__main__.StartProcessesTest) ... bar stdout from 1 2022-09-27T16:16:54.6279896Z bar stderr from 1 2022-09-27T16:16:54.7092383Z failed (exitcode: 138) local_rank: 0 (pid: 12688) of binary: distributed/elastic/multiprocessing/bin/echo1.py 2022-09-27T16:16:54.7100864Z ok (0.114s) 2022-09-27T16:16:54.7185300Z test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.008s) 2022-09-27T16:16:54.7453325Z test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last): 2022-09-27T16:16:54.7453970Z File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in 2022-09-27T16:16:54.7454254Z raise RuntimeError(f"raised from {rank}") 2022-09-27T16:16:54.7454463Z RuntimeError: raised from 0 2022-09-27T16:16:54.7488048Z bar from 1 2022-09-27T16:16:54.8304250Z failed (exitcode: 1) local_rank: 0 (pid: 12691) of binary: distributed/elastic/multiprocessing/bin/echo2.py 2022-09-27T16:16:54.8312245Z ok (0.113s) 2022-09-27T16:16:56.1721752Z test_function_large_ret_val (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:56.1722369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:56.1723556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:56.1724162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:56.1802572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:56.1803172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:56.1805792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:56.1806428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:56.1880406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:56.1881023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:56.1883647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:56.1884188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:56.1999106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:56.1999732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:56.2002719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:56.2003429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:56.8585388Z Closing process 12693 via signal SIGTERM 2022-09-27T16:16:56.8585821Z Closing process 12694 via signal SIGTERM 2022-09-27T16:16:56.8586195Z Closing process 12696 via signal SIGTERM 2022-09-27T16:16:56.8613906Z ok (2.030s) 2022-09-27T16:16:56.8629252Z test_function_raise (__main__.StartProcessesTest) 2022-09-27T16:16:58.0809521Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:58.0810118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:58.0811088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:58.0811599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:58.1098837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:58.1099393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:58.1101044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:58.1101647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:16:58.4996231Z failed (exitcode: 1) local_rank: 0 (pid: 12733) of fn: echo2 (start_method: spawn) 2022-09-27T16:16:58.4996624Z Traceback (most recent call last): 2022-09-27T16:16:58.4997139Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll 2022-09-27T16:16:58.4997487Z self._pc.join(-1) 2022-09-27T16:16:58.4997846Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join 2022-09-27T16:16:58.4998181Z raise ProcessRaisedException(msg, error_index, failed_process.pid) 2022-09-27T16:16:58.4998520Z torch.multiprocessing.spawn.ProcessRaisedException: 2022-09-27T16:16:58.4998717Z 2022-09-27T16:16:58.4998897Z -- Process 0 terminated with the following error: 2022-09-27T16:16:58.4999137Z Traceback (most recent call last): 2022-09-27T16:16:58.4999494Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap 2022-09-27T16:16:58.4999762Z fn(i, *args) 2022-09-27T16:16:58.5000140Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap 2022-09-27T16:16:58.5000412Z ret = record(fn)(*args_) 2022-09-27T16:16:58.5000818Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper 2022-09-27T16:16:58.5001117Z return f(*args, **kwargs) 2022-09-27T16:16:58.5001394Z File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2 2022-09-27T16:16:58.5001671Z raise RuntimeError(msg) 2022-09-27T16:16:58.5001864Z RuntimeError: hello 2022-09-27T16:16:58.5001978Z 2022-09-27T16:16:58.5012446Z ok (1.640s) 2022-09-27T16:16:58.5034848Z test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.002s) 2022-09-27T16:16:58.5048369Z test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.001s) 2022-09-27T16:16:58.5102538Z test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 12753 via signal SIGTERM 2022-09-27T16:16:58.5126534Z ok (0.008s) 2022-09-27T16:16:58.5156475Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn) 2022-09-27T16:16:58.5156841Z Traceback (most recent call last): 2022-09-27T16:16:58.5157269Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll 2022-09-27T16:16:58.5157575Z self._pc.join(-1) 2022-09-27T16:16:58.5157817Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__ 2022-09-27T16:16:58.5158079Z return _mock_self._mock_call(*args, **kwargs) 2022-09-27T16:16:58.5158352Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call 2022-09-27T16:16:58.5158567Z raise effect 2022-09-27T16:16:58.5158840Z torch.multiprocessing.spawn.ProcessRaisedException: test msg 2022-09-27T16:16:58.5166518Z ok (0.004s) 2022-09-27T16:16:59.6874901Z test_pcontext_wait (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:16:59.6875365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:16:59.6877012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:16:59.6877354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:01.0239477Z ok (2.507s) 2022-09-27T16:17:01.0315729Z test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 12764 closing signal SIGTERM 2022-09-27T16:17:01.0330549Z ok (0.009s) 2022-09-27T16:17:01.0350299Z test_to_map (__main__.StartProcessesTest) ... ok (0.002s) 2022-09-27T16:17:01.0358639Z test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s) 2022-09-27T16:17:02.2842145Z test_void_function (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:02.2842879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:02.2843648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:02.2844172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:02.2910333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:02.2911053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:02.2913104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:02.2913707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:02.4081608Z world 2022-09-27T16:17:02.4135578Z hello 2022-09-27T16:17:02.6913123Z Closing process 12766 via signal SIGTERM 2022-09-27T16:17:02.6932200Z ok (1.657s) 2022-09-27T16:17:02.6949907Z test_from_str_bad_input (__main__.StdTest) ... ok (0.002s) 2022-09-27T16:17:02.6961097Z test_from_value (__main__.StdTest) ... ok (0.001s) 2022-09-27T16:17:02.6970034Z test_from_value_map (__main__.StdTest) ... ok (0.001s) 2022-09-27T16:17:02.6970894Z 2022-09-27T16:17:02.6971233Z ---------------------------------------------------------------------- 2022-09-27T16:17:02.6971685Z Ran 38 tests in 19.297s 2022-09-27T16:17:02.6971898Z 2022-09-27T16:17:02.6972010Z OK 2022-09-27T16:17:02.6972158Z 2022-09-27T16:17:02.6972232Z Generating XML reports... 2022-09-27T16:17:02.7010116Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220927161643.xml 2022-09-27T16:17:02.7027105Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220927161643.xml 2022-09-27T16:17:02.7041844Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220927161643.xml 2022-09-27T16:17:02.7045534Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220927161643.xml 2022-09-27T16:17:03.0435913Z Running distributed/test_store ... [2022-09-27 16:17:03.043169] 2022-09-27T16:17:03.0436749Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:17:03.043255] 2022-09-27T16:17:04.2663408Z test_compare_set (__main__.FileStoreTest) 2022-09-27T16:17:04.2663885Z test_set_get (__main__.FileStoreTest) 2022-09-27T16:17:04.2664297Z test_compare_set (__main__.HashStoreTest) 2022-09-27T16:17:04.2664622Z test_set_get (__main__.HashStoreTest) 2022-09-27T16:17:04.2664875Z test_compare_set (__main__.PrefixFileStoreTest) 2022-09-27T16:17:04.2665112Z test_set_get (__main__.PrefixFileStoreTest) 2022-09-27T16:17:04.2665371Z test_get_underlying_store (__main__.PrefixStoreTest) 2022-09-27T16:17:04.2665632Z test_compare_set (__main__.PrefixTCPStoreTest) 2022-09-27T16:17:04.2665875Z test_set_get (__main__.PrefixTCPStoreTest) 2022-09-27T16:17:04.2666093Z test_set_get (__main__.PythonStoreTest) 2022-09-27T16:17:04.2666326Z test_nominal (__main__.RendezvousEnvTest) 2022-09-27T16:17:04.2666572Z test_common_errors (__main__.RendezvousFileTest) 2022-09-27T16:17:04.2666801Z test_nominal (__main__.RendezvousFileTest) 2022-09-27T16:17:04.2667042Z test_common_errors (__main__.RendezvousTCPTest) 2022-09-27T16:17:04.2667284Z test_dns_timeout (__main__.RendezvousTCPTest) 2022-09-27T16:17:04.2667511Z test_nominal (__main__.RendezvousTCPTest) 2022-09-27T16:17:04.2667764Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest) 2022-09-27T16:17:04.2668017Z test_unknown_handler (__main__.RendezvousTest) 2022-09-27T16:17:04.2668255Z test_url_with_node_params (__main__.RendezvousTest) 2022-09-27T16:17:04.2668504Z test_address_already_in_use (__main__.TCPStoreTest) 2022-09-27T16:17:04.2668741Z test_compare_set (__main__.TCPStoreTest) 2022-09-27T16:17:04.2668994Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) 2022-09-27T16:17:04.2669258Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) 2022-09-27T16:17:04.2669537Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) 2022-09-27T16:17:04.2669793Z test_multitenancy (__main__.TCPStoreTest) 2022-09-27T16:17:04.2670014Z test_numkeys_delkeys (__main__.TCPStoreTest) 2022-09-27T16:17:04.2670236Z test_set_get (__main__.TCPStoreTest) 2022-09-27T16:17:05.2931124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:05.2931510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:05.2941078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:05.2941470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:05.4571315Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:05.4580185Z 2022-09-27T16:17:05.4580446Z Running tests... 2022-09-27T16:17:05.4580894Z ---------------------------------------------------------------------- 2022-09-27T16:17:05.6928973Z test_compare_set (__main__.FileStoreTest) ... ok (0.235s) 2022-09-27T16:17:05.6929314Z 2022-09-27T16:17:05.6929787Z ---------------------------------------------------------------------- 2022-09-27T16:17:05.6930225Z Ran 1 test in 0.235s 2022-09-27T16:17:05.6930430Z 2022-09-27T16:17:05.6930536Z OK 2022-09-27T16:17:05.6930705Z 2022-09-27T16:17:05.6931072Z Generating XML reports... 2022-09-27T16:17:05.6953677Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220927161705.xml 2022-09-27T16:17:07.0111092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:07.0111454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:07.0119698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:07.0120240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:07.1758058Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:07.1766380Z 2022-09-27T16:17:07.1766662Z Running tests... 2022-09-27T16:17:07.1767254Z ---------------------------------------------------------------------- 2022-09-27T16:17:07.4172993Z test_set_get (__main__.FileStoreTest) ... ok (0.240s) 2022-09-27T16:17:07.4173326Z 2022-09-27T16:17:07.4173791Z ---------------------------------------------------------------------- 2022-09-27T16:17:07.4174066Z Ran 1 test in 0.241s 2022-09-27T16:17:07.4174180Z 2022-09-27T16:17:07.4174238Z OK 2022-09-27T16:17:07.4174318Z 2022-09-27T16:17:07.4174411Z Generating XML reports... 2022-09-27T16:17:07.4196476Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220927161707.xml 2022-09-27T16:17:08.7219154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:08.7219576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:08.7227704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:08.7228219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:08.8828562Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:08.8838074Z 2022-09-27T16:17:08.8838226Z Running tests... 2022-09-27T16:17:08.8838855Z ---------------------------------------------------------------------- 2022-09-27T16:17:09.1226384Z test_compare_set (__main__.HashStoreTest) ... ok (0.239s) 2022-09-27T16:17:09.1226636Z 2022-09-27T16:17:09.1227145Z ---------------------------------------------------------------------- 2022-09-27T16:17:09.1227452Z Ran 1 test in 0.239s 2022-09-27T16:17:09.1227564Z 2022-09-27T16:17:09.1227623Z OK 2022-09-27T16:17:09.1227714Z 2022-09-27T16:17:09.1227786Z Generating XML reports... 2022-09-27T16:17:09.1250038Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220927161708.xml 2022-09-27T16:17:10.4325716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:10.4326364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:10.4334660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:10.4335280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:10.5929206Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:10.5939060Z 2022-09-27T16:17:10.5939478Z Running tests... 2022-09-27T16:17:10.5939890Z ---------------------------------------------------------------------- 2022-09-27T16:17:10.8288143Z test_set_get (__main__.HashStoreTest) ... ok (0.235s) 2022-09-27T16:17:10.8288422Z 2022-09-27T16:17:10.8288856Z ---------------------------------------------------------------------- 2022-09-27T16:17:10.8289292Z Ran 1 test in 0.235s 2022-09-27T16:17:10.8289500Z 2022-09-27T16:17:10.8289607Z OK 2022-09-27T16:17:10.8289772Z 2022-09-27T16:17:10.8289934Z Generating XML reports... 2022-09-27T16:17:10.8315491Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220927161710.xml 2022-09-27T16:17:12.1330985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:12.1331943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:12.1340166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:12.1340779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:12.2915451Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:12.2925355Z 2022-09-27T16:17:12.2925802Z Running tests... 2022-09-27T16:17:12.2926238Z ---------------------------------------------------------------------- 2022-09-27T16:17:12.5359074Z test_compare_set (__main__.PrefixFileStoreTest) ... ok (0.243s) 2022-09-27T16:17:12.5359426Z 2022-09-27T16:17:12.5359833Z ---------------------------------------------------------------------- 2022-09-27T16:17:12.5360066Z Ran 1 test in 0.243s 2022-09-27T16:17:12.5360180Z 2022-09-27T16:17:12.5360248Z OK 2022-09-27T16:17:12.5360340Z 2022-09-27T16:17:12.5360426Z Generating XML reports... 2022-09-27T16:17:12.5382622Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220927161712.xml 2022-09-27T16:17:13.8329973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:13.8330379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:13.8338390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:13.8338878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:13.9937753Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:13.9946828Z 2022-09-27T16:17:13.9946948Z Running tests... 2022-09-27T16:17:13.9947368Z ---------------------------------------------------------------------- 2022-09-27T16:17:14.2332730Z test_set_get (__main__.PrefixFileStoreTest) ... ok (0.238s) 2022-09-27T16:17:14.2332930Z 2022-09-27T16:17:14.2333229Z ---------------------------------------------------------------------- 2022-09-27T16:17:14.2333475Z Ran 1 test in 0.238s 2022-09-27T16:17:14.2333590Z 2022-09-27T16:17:14.2333655Z OK 2022-09-27T16:17:14.2333801Z 2022-09-27T16:17:14.2333874Z Generating XML reports... 2022-09-27T16:17:14.2358091Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220927161713.xml 2022-09-27T16:17:15.5484486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:15.5484855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:15.5492128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:15.5492636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:15.7078320Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:15.7087433Z 2022-09-27T16:17:15.7087569Z Running tests... 2022-09-27T16:17:15.7088165Z ---------------------------------------------------------------------- 2022-09-27T16:17:15.7112427Z test_get_underlying_store (__main__.PrefixStoreTest) ... ok (0.002s) 2022-09-27T16:17:15.7112908Z 2022-09-27T16:17:15.7113228Z ---------------------------------------------------------------------- 2022-09-27T16:17:15.7113547Z Ran 1 test in 0.003s 2022-09-27T16:17:15.7113696Z 2022-09-27T16:17:15.7113758Z OK 2022-09-27T16:17:15.7113861Z 2022-09-27T16:17:15.7114277Z Generating XML reports... 2022-09-27T16:17:15.7138408Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20220927161715.xml 2022-09-27T16:17:16.9572599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:16.9573332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:16.9582115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:16.9582720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:17.1202741Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:17.1212656Z 2022-09-27T16:17:17.1212938Z Running tests... 2022-09-27T16:17:17.1213522Z ---------------------------------------------------------------------- 2022-09-27T16:17:17.3589885Z test_compare_set (__main__.PrefixTCPStoreTest) ... ok (0.237s) 2022-09-27T16:17:17.3590201Z 2022-09-27T16:17:17.3590731Z ---------------------------------------------------------------------- 2022-09-27T16:17:17.3591210Z Ran 1 test in 0.238s 2022-09-27T16:17:17.3591412Z 2022-09-27T16:17:17.3591505Z OK 2022-09-27T16:17:17.3591668Z 2022-09-27T16:17:17.3591824Z Generating XML reports... 2022-09-27T16:17:17.3616009Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220927161717.xml 2022-09-27T16:17:18.6601410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:18.6601998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:18.6610321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:18.6610928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:18.8210528Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:18.8220436Z 2022-09-27T16:17:18.8220903Z Running tests... 2022-09-27T16:17:18.8221522Z ---------------------------------------------------------------------- 2022-09-27T16:17:19.0603722Z test_set_get (__main__.PrefixTCPStoreTest) ... ok (0.238s) 2022-09-27T16:17:19.0604079Z 2022-09-27T16:17:19.0604446Z ---------------------------------------------------------------------- 2022-09-27T16:17:19.0604713Z Ran 1 test in 0.238s 2022-09-27T16:17:19.0604816Z 2022-09-27T16:17:19.0604875Z OK 2022-09-27T16:17:19.0604966Z 2022-09-27T16:17:19.0605051Z Generating XML reports... 2022-09-27T16:17:19.0629121Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220927161718.xml 2022-09-27T16:17:20.3744371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:20.3744753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:20.3752655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:20.3753036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:20.5332147Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:20.5340684Z 2022-09-27T16:17:20.5340819Z Running tests... 2022-09-27T16:17:20.5341370Z ---------------------------------------------------------------------- 2022-09-27T16:17:20.7668149Z test_set_get (__main__.PythonStoreTest) ... ok (0.232s) 2022-09-27T16:17:20.7668368Z 2022-09-27T16:17:20.7668713Z ---------------------------------------------------------------------- 2022-09-27T16:17:20.7668960Z Ran 1 test in 0.233s 2022-09-27T16:17:20.7669074Z 2022-09-27T16:17:20.7669135Z OK 2022-09-27T16:17:20.7669228Z 2022-09-27T16:17:20.7669316Z Generating XML reports... 2022-09-27T16:17:20.7692978Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220927161720.xml 2022-09-27T16:17:22.0492480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:22.0492866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:22.0500024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:22.0500485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:22.2094384Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:22.2103464Z 2022-09-27T16:17:22.2103544Z Running tests... 2022-09-27T16:17:22.2103982Z ---------------------------------------------------------------------- 2022-09-27T16:17:22.4491660Z test_nominal (__main__.RendezvousEnvTest) ... ok (0.239s) 2022-09-27T16:17:22.4491909Z 2022-09-27T16:17:22.4492227Z ---------------------------------------------------------------------- 2022-09-27T16:17:22.4492476Z Ran 1 test in 0.239s 2022-09-27T16:17:22.4492592Z 2022-09-27T16:17:22.4492654Z OK 2022-09-27T16:17:22.4492762Z 2022-09-27T16:17:22.4492858Z Generating XML reports... 2022-09-27T16:17:22.4517450Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220927161722.xml 2022-09-27T16:17:23.7469977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:23.7470353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:23.7478669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:23.7479034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:23.9045286Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:23.9054555Z 2022-09-27T16:17:23.9054667Z Running tests... 2022-09-27T16:17:23.9055256Z ---------------------------------------------------------------------- 2022-09-27T16:17:24.1381354Z test_common_errors (__main__.RendezvousFileTest) ... ok (0.232s) 2022-09-27T16:17:24.1381769Z 2022-09-27T16:17:24.1382055Z ---------------------------------------------------------------------- 2022-09-27T16:17:24.1382330Z Ran 1 test in 0.233s 2022-09-27T16:17:24.1382450Z 2022-09-27T16:17:24.1382510Z OK 2022-09-27T16:17:24.1382601Z 2022-09-27T16:17:24.1382686Z Generating XML reports... 2022-09-27T16:17:24.1406360Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220927161723.xml 2022-09-27T16:17:25.4229831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:25.4230207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:25.4238379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:25.4238840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:25.5822211Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:25.5830393Z 2022-09-27T16:17:25.5830527Z Running tests... 2022-09-27T16:17:25.5831206Z ---------------------------------------------------------------------- 2022-09-27T16:17:25.8170573Z test_nominal (__main__.RendezvousFileTest) ... ok (0.234s) 2022-09-27T16:17:25.8170897Z 2022-09-27T16:17:25.8171314Z ---------------------------------------------------------------------- 2022-09-27T16:17:25.8171562Z Ran 1 test in 0.234s 2022-09-27T16:17:25.8171677Z 2022-09-27T16:17:25.8171725Z OK 2022-09-27T16:17:25.8171820Z 2022-09-27T16:17:25.8171916Z Generating XML reports... 2022-09-27T16:17:25.8193941Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220927161725.xml 2022-09-27T16:17:27.1021893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:27.1022551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:27.1030157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:27.1030852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:27.2620898Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:27.2629522Z 2022-09-27T16:17:27.2629648Z Running tests... 2022-09-27T16:17:27.2630195Z ---------------------------------------------------------------------- 2022-09-27T16:17:27.4999828Z test_common_errors (__main__.RendezvousTCPTest) ... ok (0.237s) 2022-09-27T16:17:27.5000171Z 2022-09-27T16:17:27.5000562Z ---------------------------------------------------------------------- 2022-09-27T16:17:27.5000807Z Ran 1 test in 0.237s 2022-09-27T16:17:27.5000926Z 2022-09-27T16:17:27.5000972Z OK 2022-09-27T16:17:27.5001062Z 2022-09-27T16:17:27.5001157Z Generating XML reports... 2022-09-27T16:17:27.5023771Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927161727.xml 2022-09-27T16:17:28.8267308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:28.8267679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:28.8276900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:28.8277263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:28.9911434Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:28.9921851Z 2022-09-27T16:17:28.9921994Z Running tests... 2022-09-27T16:17:28.9922492Z ---------------------------------------------------------------------- 2022-09-27T16:17:29.2387219Z test_dns_timeout (__main__.RendezvousTCPTest) ... [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known). 2022-09-27T16:17:29.2387685Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456). 2022-09-27T16:17:29.2389332Z ok (0.247s) 2022-09-27T16:17:29.2390086Z 2022-09-27T16:17:29.2390728Z ---------------------------------------------------------------------- 2022-09-27T16:17:29.2391199Z Ran 1 test in 0.247s 2022-09-27T16:17:29.2391326Z 2022-09-27T16:17:29.2391387Z OK 2022-09-27T16:17:29.2391481Z 2022-09-27T16:17:29.2391561Z Generating XML reports... 2022-09-27T16:17:29.2417313Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927161728.xml 2022-09-27T16:17:30.5449966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:30.5450342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:30.5458673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:30.5459042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:30.7110284Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:30.7123014Z 2022-09-27T16:17:30.7123122Z Running tests... 2022-09-27T16:17:30.7123663Z ---------------------------------------------------------------------- 2022-09-27T16:17:30.9557406Z test_nominal (__main__.RendezvousTCPTest) ... ok (0.243s) 2022-09-27T16:17:30.9557591Z 2022-09-27T16:17:30.9557862Z ---------------------------------------------------------------------- 2022-09-27T16:17:30.9558109Z Ran 1 test in 0.243s 2022-09-27T16:17:30.9558212Z 2022-09-27T16:17:30.9558275Z OK 2022-09-27T16:17:30.9558369Z 2022-09-27T16:17:30.9558454Z Generating XML reports... 2022-09-27T16:17:30.9583216Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927161730.xml 2022-09-27T16:17:32.2668788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:32.2669741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:32.2678298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:32.2678898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:32.4277061Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:32.4286900Z 2022-09-27T16:17:32.4287181Z Running tests... 2022-09-27T16:17:32.4287825Z ---------------------------------------------------------------------- 2022-09-27T16:17:42.8631990Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... ok (10.434s) 2022-09-27T16:17:42.8632348Z 2022-09-27T16:17:42.8632859Z ---------------------------------------------------------------------- 2022-09-27T16:17:42.8633127Z Ran 1 test in 10.434s 2022-09-27T16:17:42.8633248Z 2022-09-27T16:17:42.8633307Z OK 2022-09-27T16:17:42.8633397Z 2022-09-27T16:17:42.8633482Z Generating XML reports... 2022-09-27T16:17:42.8660172Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927161732.xml 2022-09-27T16:17:44.2008628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:44.2009001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:44.2016986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:44.2017350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:44.3608336Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:44.3618424Z 2022-09-27T16:17:44.3618758Z Running tests... 2022-09-27T16:17:44.3619121Z ---------------------------------------------------------------------- 2022-09-27T16:17:44.5949374Z test_unknown_handler (__main__.RendezvousTest) ... ok (0.233s) 2022-09-27T16:17:44.5949697Z 2022-09-27T16:17:44.5950169Z ---------------------------------------------------------------------- 2022-09-27T16:17:44.5950435Z Ran 1 test in 0.233s 2022-09-27T16:17:44.5950538Z 2022-09-27T16:17:44.5950596Z OK 2022-09-27T16:17:44.5950753Z 2022-09-27T16:17:44.5950845Z Generating XML reports... 2022-09-27T16:17:44.5974186Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220927161744.xml 2022-09-27T16:17:45.9157398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:45.9157814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:45.9166203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:45.9166698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:46.0755339Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:46.0764421Z 2022-09-27T16:17:46.0764564Z Running tests... 2022-09-27T16:17:46.0765185Z ---------------------------------------------------------------------- 2022-09-27T16:17:46.3168381Z test_url_with_node_params (__main__.RendezvousTest) ... ok (0.240s) 2022-09-27T16:17:46.3168740Z 2022-09-27T16:17:46.3169211Z ---------------------------------------------------------------------- 2022-09-27T16:17:46.3169630Z Ran 1 test in 0.240s 2022-09-27T16:17:46.3169837Z 2022-09-27T16:17:46.3169943Z OK 2022-09-27T16:17:46.3170108Z 2022-09-27T16:17:46.3170260Z Generating XML reports... 2022-09-27T16:17:46.3194394Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220927161746.xml 2022-09-27T16:17:47.6255774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:47.6256717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:47.6264921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:47.6265291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:47.7851508Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:47.7860121Z 2022-09-27T16:17:47.7860354Z Running tests... 2022-09-27T16:17:47.7860810Z ---------------------------------------------------------------------- 2022-09-27T16:17:48.0253212Z test_address_already_in_use (__main__.TCPStoreTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:42545 (errno: 98 - Address already in use). 2022-09-27T16:17:48.0264667Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:42545 (errno: 98 - Address already in use). 2022-09-27T16:17:48.0265105Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address. 2022-09-27T16:17:48.0268177Z ok (0.241s) 2022-09-27T16:17:48.0268505Z 2022-09-27T16:17:48.0268869Z ---------------------------------------------------------------------- 2022-09-27T16:17:48.0269114Z Ran 1 test in 0.241s 2022-09-27T16:17:48.0269228Z 2022-09-27T16:17:48.0269288Z OK 2022-09-27T16:17:48.0269378Z 2022-09-27T16:17:48.0269463Z Generating XML reports... 2022-09-27T16:17:48.0294329Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161747.xml 2022-09-27T16:17:49.3392851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:49.3393219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:49.3401012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:49.3401382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:49.4978810Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:49.4986886Z 2022-09-27T16:17:49.4986991Z Running tests... 2022-09-27T16:17:49.4987960Z ---------------------------------------------------------------------- 2022-09-27T16:17:49.7355785Z test_compare_set (__main__.TCPStoreTest) ... ok (0.237s) 2022-09-27T16:17:49.7356158Z 2022-09-27T16:17:49.7356460Z ---------------------------------------------------------------------- 2022-09-27T16:17:49.7356710Z Ran 1 test in 0.237s 2022-09-27T16:17:49.7356822Z 2022-09-27T16:17:49.7356870Z OK 2022-09-27T16:17:49.7356960Z 2022-09-27T16:17:49.7357044Z Generating XML reports... 2022-09-27T16:17:49.7379172Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161749.xml 2022-09-27T16:17:51.0291113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:51.0291569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:51.0298875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:51.0299313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:51.1890773Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:51.1899704Z 2022-09-27T16:17:51.1899808Z Running tests... 2022-09-27T16:17:51.1900342Z ---------------------------------------------------------------------- 2022-09-27T16:17:51.4275631Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:17:51.4276513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-09-27T16:17:51.4903356Z ok (0.300s) 2022-09-27T16:17:51.4903575Z 2022-09-27T16:17:51.4904043Z ---------------------------------------------------------------------- 2022-09-27T16:17:51.4904498Z Ran 1 test in 0.300s 2022-09-27T16:17:51.4904684Z 2022-09-27T16:17:51.4904743Z OK 2022-09-27T16:17:51.4904837Z 2022-09-27T16:17:51.4904933Z Generating XML reports... 2022-09-27T16:17:51.4931024Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161751.xml 2022-09-27T16:17:52.8048946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:52.8049331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:52.8057590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:52.8057985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:52.9629478Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:52.9638228Z 2022-09-27T16:17:52.9638381Z Running tests... 2022-09-27T16:17:52.9638982Z ---------------------------------------------------------------------- 2022-09-27T16:17:53.2133583Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... ok (0.249s) 2022-09-27T16:17:53.2133991Z 2022-09-27T16:17:53.2134359Z ---------------------------------------------------------------------- 2022-09-27T16:17:53.2134617Z Ran 1 test in 0.249s 2022-09-27T16:17:53.2134731Z 2022-09-27T16:17:53.2134791Z OK 2022-09-27T16:17:53.2134870Z 2022-09-27T16:17:53.2134957Z Generating XML reports... 2022-09-27T16:17:53.2179751Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161752.xml 2022-09-27T16:17:54.5237665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:54.5238050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:54.5246463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:54.5246848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:54.6859616Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:54.6868266Z 2022-09-27T16:17:54.6868412Z Running tests... 2022-09-27T16:17:54.6868983Z ---------------------------------------------------------------------- 2022-09-27T16:17:54.9300069Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... ok (0.243s) 2022-09-27T16:17:54.9300357Z 2022-09-27T16:17:54.9300620Z ---------------------------------------------------------------------- 2022-09-27T16:17:54.9301314Z Ran 1 test in 0.243s 2022-09-27T16:17:54.9301535Z 2022-09-27T16:17:54.9301631Z OK 2022-09-27T16:17:54.9301796Z 2022-09-27T16:17:54.9301963Z Generating XML reports... 2022-09-27T16:17:54.9326030Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161754.xml 2022-09-27T16:17:56.2337367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:56.2337773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:56.2345715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:56.2346122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:56.3898213Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:56.3907803Z 2022-09-27T16:17:56.3908045Z Running tests... 2022-09-27T16:17:56.3908937Z ---------------------------------------------------------------------- 2022-09-27T16:17:56.6280246Z test_multitenancy (__main__.TCPStoreTest) ... ok (0.237s) 2022-09-27T16:17:56.6280851Z 2022-09-27T16:17:56.6281316Z ---------------------------------------------------------------------- 2022-09-27T16:17:56.6281769Z Ran 1 test in 0.237s 2022-09-27T16:17:56.6281974Z 2022-09-27T16:17:56.6282068Z OK 2022-09-27T16:17:56.6282231Z 2022-09-27T16:17:56.6282386Z Generating XML reports... 2022-09-27T16:17:56.6306148Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161756.xml 2022-09-27T16:17:57.9503654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:17:57.9504298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:17:57.9512651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:17:57.9513279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:17:58.1081319Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:17:58.1090752Z 2022-09-27T16:17:58.1091137Z Running tests... 2022-09-27T16:17:58.1091538Z ---------------------------------------------------------------------- 2022-09-27T16:18:00.3679056Z test_numkeys_delkeys (__main__.TCPStoreTest) ... ok (2.259s) 2022-09-27T16:18:00.3679349Z 2022-09-27T16:18:00.3679845Z ---------------------------------------------------------------------- 2022-09-27T16:18:00.3680253Z Ran 1 test in 2.259s 2022-09-27T16:18:00.3680370Z 2022-09-27T16:18:00.3680431Z OK 2022-09-27T16:18:00.3680525Z 2022-09-27T16:18:00.3680613Z Generating XML reports... 2022-09-27T16:18:00.3706055Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161758.xml 2022-09-27T16:18:01.6924956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:01.6925739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:01.6926543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:01.6927094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:01.8574883Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-09-27T16:18:01.8583706Z 2022-09-27T16:18:01.8584110Z Running tests... 2022-09-27T16:18:01.8584523Z ---------------------------------------------------------------------- 2022-09-27T16:18:02.1096090Z test_set_get (__main__.TCPStoreTest) ... ok (0.251s) 2022-09-27T16:18:02.1096320Z 2022-09-27T16:18:02.1096696Z ---------------------------------------------------------------------- 2022-09-27T16:18:02.1096954Z Ran 1 test in 0.251s 2022-09-27T16:18:02.1097071Z 2022-09-27T16:18:02.1097131Z OK 2022-09-27T16:18:02.1097224Z 2022-09-27T16:18:02.1097320Z Generating XML reports... 2022-09-27T16:18:02.1120966Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161801.xml 2022-09-27T16:18:02.6450157Z Running distributed/algorithms/test_join ... [2022-09-27 16:18:02.644629] 2022-09-27T16:18:02.6450972Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:02.644711] 2022-09-27T16:18:03.8572161Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join 2022-09-27T16:18:03.8583971Z 2022-09-27T16:18:03.8584102Z Running tests... 2022-09-27T16:18:03.8584501Z ---------------------------------------------------------------------- 2022-09-27T16:18:03.8593100Z test_join_kwargs (__main__.TestJoin) 2022-09-27T16:18:04.1009534Z Tests passing keyword arguments to the context manager. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13157 2022-09-27T16:18:04.1037289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13158 2022-09-27T16:18:05.1320982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:05.1321871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:05.1659816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:05.1660463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:05.1898393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:05.1899057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:05.2251191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:05.2251818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:05.2893991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:05.3499000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:05.3706716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:05.3707158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:05.3707821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:05.3708374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:05.7073149Z ok (1.849s) 2022-09-27T16:18:05.7080830Z test_multiple_joinable_disable (__main__.TestJoin) 2022-09-27T16:18:05.7125014Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13185 2022-09-27T16:18:05.7158132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13186 2022-09-27T16:18:06.7356921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:06.7357287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:06.7427537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:06.7427913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:06.7717163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:06.7717534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:06.7783437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:06.7783943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:06.8940262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:06.9009171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:06.9116776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:06.9117268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:06.9117877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:06.9118400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:07.2189505Z ok (1.511s) 2022-09-27T16:18:07.2197798Z test_multiple_joinables (__main__.TestJoin) 2022-09-27T16:18:07.2242077Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13213 2022-09-27T16:18:07.2274728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13214 2022-09-27T16:18:08.2623415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:08.2623806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:08.2965834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:08.2966203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:08.2984510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:08.2984904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:08.3325634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:08.3326003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:08.4222080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:08.4557467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:08.4732520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:08.4732995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:08.4733613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:08.4734157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:08.8307917Z ok (1.612s) 2022-09-27T16:18:08.8314034Z test_multiple_joinables_throw (__main__.TestJoin) 2022-09-27T16:18:08.8358523Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13241 2022-09-27T16:18:08.8392775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13242 2022-09-27T16:18:09.8838466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:09.8839113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:09.8854251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:09.8854912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:09.9209237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:09.9209675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:09.9225648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:09.9226052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:10.0437239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:10.0453532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:10.0644736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:10.0645427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:10.0646421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:10.0646973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:10.3423981Z ok (1.511s) 2022-09-27T16:18:10.3431841Z test_single_joinable (__main__.TestJoin) 2022-09-27T16:18:10.3478241Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13269 2022-09-27T16:18:10.3511251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13270 2022-09-27T16:18:11.3759064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:11.3759658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:11.3785294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:11.3785982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:11.4123653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:11.4124437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:11.4152267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:11.4152844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:11.5355203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:11.5390168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:11.5599073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:11.5599508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:11.5600176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:11.5600765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:11.8542330Z ok (1.512s) 2022-09-27T16:18:11.8549199Z test_single_joinable_disable (__main__.TestJoin) 2022-09-27T16:18:11.8593043Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13297 2022-09-27T16:18:11.8625783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13298 2022-09-27T16:18:12.9049290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:12.9049683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:12.9264908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:12.9265385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:12.9395773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:12.9396302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:12.9602560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:12.9603091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:13.0633911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:13.0834198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:13.0941672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:13.0942375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:13.0943231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:13.0943760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:13.3657088Z ok (1.511s) 2022-09-27T16:18:13.3664726Z test_single_joinable_main_hooks (__main__.TestJoin) 2022-09-27T16:18:13.3710066Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13325 2022-09-27T16:18:13.3743000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13326 2022-09-27T16:18:14.4214848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:14.4215495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:14.4247975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:14.4563922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:14.4564946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:14.4565645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:14.4599934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:14.4600535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:14.5791157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:14.5823895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:14.6032254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:14.6032941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:14.6033556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:14.6034087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:14.8773409Z ok (1.512s) 2022-09-27T16:18:14.8779460Z test_single_joinable_post_hooks (__main__.TestJoin) 2022-09-27T16:18:14.8824666Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13353 2022-09-27T16:18:14.8858601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13354 2022-09-27T16:18:15.9246188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:15.9246866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:15.9333949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:15.9334482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:15.9614542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:15.9615127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:15.9706369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:15.9707029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:16.0845758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:16.0927979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:16.1035660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:16.1036250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:16.1037182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:16.1038003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:16.3890465Z ok (1.512s) 2022-09-27T16:18:16.3896636Z test_single_joinable_throw (__main__.TestJoin) 2022-09-27T16:18:16.3942252Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13381 2022-09-27T16:18:16.3976177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13382 2022-09-27T16:18:17.4288707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:17.4289134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:17.4349678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:17.4350023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:17.4644545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:17.4644912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:17.4707112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:17.4707498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:17.5861666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:17.5929202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:17.6036778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:17.6037527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:17.6038161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:17.6038694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:18:17.9010529Z ok (1.512s) 2022-09-27T16:18:17.9010844Z 2022-09-27T16:18:17.9011330Z ---------------------------------------------------------------------- 2022-09-27T16:18:17.9011617Z Ran 9 tests in 14.043s 2022-09-27T16:18:17.9011740Z 2022-09-27T16:18:17.9011803Z OK 2022-09-27T16:18:17.9011889Z 2022-09-27T16:18:17.9011987Z Generating XML reports... 2022-09-27T16:18:17.9052637Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220927161803.xml 2022-09-27T16:18:18.2094446Z Running distributed/fsdp/test_fsdp_exec_order ... [2022-09-27 16:18:18.209057] 2022-09-27T16:18:18.2095040Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:18.209143] 2022-09-27T16:18:19.4418179Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order 2022-09-27T16:18:19.4429158Z 2022-09-27T16:18:19.4429251Z Running tests... 2022-09-27T16:18:19.4430266Z ---------------------------------------------------------------------- 2022-09-27T16:18:19.4438789Z test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) 2022-09-27T16:18:19.6934184Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13420 2022-09-27T16:18:19.6962250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13421 2022-09-27T16:18:19.6989479Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13422 2022-09-27T16:18:19.7019724Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13423 2022-09-27T16:18:20.8898234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:20.8898956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:20.8901990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:20.8902627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:20.8944288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:20.8944924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:20.8950151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:20.8950815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:20.9218871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:20.9219511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:20.9223268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:20.9223879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:20.9386413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:20.9387038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:20.9391631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:20.9392240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:21.0212510Z dist init r=0, world=4 2022-09-27T16:18:21.0235793Z dist init r=1, world=4 2022-09-27T16:18:21.0566364Z dist init r=2, world=4 2022-09-27T16:18:21.0669556Z dist init r=3, world=4 2022-09-27T16:18:21.0779024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:21.0877248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:21.0877798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:21.0878319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:21.0878938Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:21.0879465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:21.0880007Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:21.0880717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:21.0985093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:21.0985859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:21.0986418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:21.0987034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:21.4065549Z skip: Need at least 2 CUDA devices (1.963s) 2022-09-27T16:18:21.4073851Z test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) 2022-09-27T16:18:21.4119674Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13476 2022-09-27T16:18:21.4154405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13477 2022-09-27T16:18:21.4183303Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13478 2022-09-27T16:18:21.4212295Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13479 2022-09-27T16:18:22.5448921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:22.5449425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:22.5452835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:22.5453548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:22.5805209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:22.5805916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:22.5808551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:22.5809253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:22.5876651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:22.5877262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:22.5881119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:22.5881775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:22.6025537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:22.6026164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:22.6030546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:22.6031280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:22.6742966Z dist init r=0, world=4 2022-09-27T16:18:22.7092484Z dist init r=1, world=4 2022-09-27T16:18:22.7164920Z dist init r=3, world=4 2022-09-27T16:18:22.7299997Z dist init r=2, world=4 2022-09-27T16:18:22.7475115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:22.7556207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:22.7658677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:22.7659393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:22.7660092Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:22.7660609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:22.7661310Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:22.7678269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:22.7767715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:22.7768339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:22.7768851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:22.7769195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:23.1252987Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:18:23.1268395Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder) 2022-09-27T16:18:23.1314367Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13532 2022-09-27T16:18:23.1348930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13533 2022-09-27T16:18:23.1377090Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13534 2022-09-27T16:18:23.1405654Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13535 2022-09-27T16:18:24.2440199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:24.2440713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:24.2444126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:24.2444650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:24.2475306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:24.2475946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:24.2480637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:24.2481309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:24.2582858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:24.2583441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:24.2588162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:24.2588805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:24.2894781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:24.2895402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:24.2898530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:24.2899159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:24.3712783Z dist init r=1, world=4 2022-09-27T16:18:24.3747517Z dist init r=0, world=4 2022-09-27T16:18:24.3845115Z dist init r=3, world=4 2022-09-27T16:18:24.4148009Z dist init r=2, world=4 2022-09-27T16:18:24.4360656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:24.4361054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:24.4463004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:24.4463598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:24.4464596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:24.4465248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:24.4465758Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:24.4466279Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:24.4570709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:24.4571304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:24.4571768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:24.4572123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:24.7445927Z skip: Need at least 2 CUDA devices (1.619s) 2022-09-27T16:18:24.7460515Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder) 2022-09-27T16:18:24.7505506Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13588 2022-09-27T16:18:24.7538997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13589 2022-09-27T16:18:24.7566596Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13590 2022-09-27T16:18:24.7595423Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13591 2022-09-27T16:18:25.8931475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:25.8932588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:25.8973011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:25.8973787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:25.9242927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:25.9243509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:25.9247139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:25.9247750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:25.9957973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:25.9958600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:25.9961840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:25.9962473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:26.0070569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:26.0071254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:26.0075411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:26.0076020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:26.0206012Z dist init r=1, world=4 2022-09-27T16:18:26.0511955Z dist init r=2, world=4 2022-09-27T16:18:26.1223582Z dist init r=0, world=4 2022-09-27T16:18:26.1336335Z dist init r=3, world=4 2022-09-27T16:18:26.1526950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:26.1629224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:26.1730169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:26.1743570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:26.1744220Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:26.1834384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:26.1835129Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:26.1835652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:26.1851142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:26.1851683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:26.1852239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:26.1852743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:26.4636509Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:18:26.4651833Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder) 2022-09-27T16:18:26.4697535Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13644 2022-09-27T16:18:26.4730303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13645 2022-09-27T16:18:26.4758190Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13646 2022-09-27T16:18:26.4786533Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13647 2022-09-27T16:18:27.6107522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:27.6108049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:27.6112324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:27.6112972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:27.6273718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:27.6274356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:27.6278791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:27.6279521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:27.6489066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:27.6489700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:27.6490436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:27.6491017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:27.6492033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:27.6492620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:27.6495576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:27.6496463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:27.7389615Z dist init r=2, world=4 2022-09-27T16:18:27.7561977Z dist init r=1, world=4 2022-09-27T16:18:27.7752011Z dist init r=3, world=4 2022-09-27T16:18:27.7766027Z dist init r=0, world=4 2022-09-27T16:18:27.7971141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:27.8102512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:27.8103003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:27.8103417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:27.8104021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:27.8104562Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:27.8105069Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:27.8174716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:27.8209625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:27.8210284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:27.8210805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:27.8211327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:28.1828774Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:18:28.1844115Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder) 2022-09-27T16:18:28.1888828Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13700 2022-09-27T16:18:28.1922844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13701 2022-09-27T16:18:28.1950943Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13702 2022-09-27T16:18:28.1979643Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13703 2022-09-27T16:18:29.3091928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:29.3092574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:29.3096246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:29.3096981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:29.3140062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:29.3140997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:29.3145364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:29.3145874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:29.3472642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:29.3473390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:29.3475590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:29.3476176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:29.3687019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:29.3687771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:29.3690879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:29.3691551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:29.4379054Z dist init r=0, world=4 2022-09-27T16:18:29.4422518Z dist init r=2, world=4 2022-09-27T16:18:29.4763441Z dist init r=3, world=4 2022-09-27T16:18:29.4980366Z dist init r=1, world=4 2022-09-27T16:18:29.5174797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:29.5274816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:29.5376964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:29.5377601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:29.5378275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:29.5378803Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:29.5379437Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:29.5380000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:29.5484579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:29.5485024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:29.5485515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:29.5486040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:29.9022272Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:18:29.9079056Z test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13756 2022-09-27T16:18:29.9113566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13757 2022-09-27T16:18:29.9142286Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13758 2022-09-27T16:18:29.9171143Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13759 2022-09-27T16:18:31.0204361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:31.0205040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:31.0205710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:31.0206289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:31.0207831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:31.0208363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:31.0210515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:31.0211080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:31.0452871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:31.0453781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:31.0455885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:31.0456643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:31.0827387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:31.0828018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:31.0831414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:31.0832047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:31.1514782Z dist init r=3, world=4 2022-09-27T16:18:31.1515141Z dist init r=1, world=4 2022-09-27T16:18:31.1749368Z dist init r=0, world=4 2022-09-27T16:18:31.2123138Z dist init r=2, world=4 2022-09-27T16:18:31.2328678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:31.2429019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:31.2531110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:31.2531695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:31.2532620Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:31.2533188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:31.2533712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:31.2534239Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:31.2639779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:31.2640374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:31.2640916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:31.2641484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:31.6212703Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:18:31.6271568Z test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13812 2022-09-27T16:18:31.6306614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13813 2022-09-27T16:18:31.6335576Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13814 2022-09-27T16:18:31.6364398Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13815 2022-09-27T16:18:32.7638672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:32.7639200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:32.7642375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:32.7642993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:32.7831758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:32.7832356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:32.7835184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:32.7836067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:32.7969537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:32.7970391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:32.7973659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:32.7974289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:32.7998196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:32.7998754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:32.8003642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:32.8004240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:32.8922574Z dist init r=2, world=4 2022-09-27T16:18:32.9112481Z dist init r=0, world=4 2022-09-27T16:18:32.9250635Z dist init r=1, world=4 2022-09-27T16:18:32.9287519Z dist init r=3, world=4 2022-09-27T16:18:32.9564950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:32.9565600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:32.9566179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:32.9566797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:32.9567410Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:32.9567955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:32.9568544Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:32.9569253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:32.9572301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:32.9572856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:32.9575482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:32.9576297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:33.2404926Z skip: Need at least 2 CUDA devices (1.619s) 2022-09-27T16:18:33.2405114Z 2022-09-27T16:18:33.2405507Z ---------------------------------------------------------------------- 2022-09-27T16:18:33.2405785Z Ran 8 tests in 13.797s 2022-09-27T16:18:33.2405900Z 2022-09-27T16:18:33.2405973Z OK (skipped=8) 2022-09-27T16:18:33.2406067Z 2022-09-27T16:18:33.2406161Z Generating XML reports... 2022-09-27T16:18:33.2446729Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20220927161819.xml 2022-09-27T16:18:33.5766286Z Running distributed/fsdp/test_fsdp_freezing_weights ... [2022-09-27 16:18:33.576203] 2022-09-27T16:18:33.5766891Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:33.576288] 2022-09-27T16:18:34.8220309Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights 2022-09-27T16:18:34.8231945Z 2022-09-27T16:18:34.8232063Z Running tests... 2022-09-27T16:18:34.8232433Z ---------------------------------------------------------------------- 2022-09-27T16:18:35.0759852Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13879 2022-09-27T16:18:35.0786282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13880 2022-09-27T16:18:35.0813720Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13881 2022-09-27T16:18:35.0843529Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13882 2022-09-27T16:18:36.2722051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:36.2722695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:36.2726269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:36.2726883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:36.3335736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:36.3336391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:36.3337111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:36.3337626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:36.3340147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:36.3340760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:36.3341506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:36.3342058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:36.3473825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:36.3474466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:36.3478794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:36.3479414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:36.4005778Z dist init r=3, world=4 2022-09-27T16:18:36.4623208Z dist init r=1, world=4 2022-09-27T16:18:36.4631725Z dist init r=0, world=4 2022-09-27T16:18:36.4740977Z dist init r=2, world=4 2022-09-27T16:18:36.5033602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:36.5135732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:36.5136444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:36.5137334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:36.5137825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:36.5138518Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:36.5139182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:36.5139696Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:36.5143627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:36.5144184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:36.5144937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:36.5145735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:36.8890720Z skip: Need at least 2 CUDA devices (2.066s) 2022-09-27T16:18:36.8944898Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13935 2022-09-27T16:18:36.8978669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13936 2022-09-27T16:18:36.9007256Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13937 2022-09-27T16:18:36.9035608Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13938 2022-09-27T16:18:38.0301559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:38.0302075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:38.0304997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:38.0305507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:38.0751703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:38.0752314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:38.0755361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:38.0755945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:38.0939454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:38.0939986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:38.0943889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:38.0944532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:38.1405876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:38.1406562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:38.1409133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:38.1409786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:38.1604353Z dist init r=0, world=4 2022-09-27T16:18:38.2036770Z dist init r=1, world=4 2022-09-27T16:18:38.2215251Z dist init r=3, world=4 2022-09-27T16:18:38.2676288Z dist init r=2, world=4 2022-09-27T16:18:38.2922217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:38.3022501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:38.3125636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:38.3126244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:38.3127225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:38.3127934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:38.3128458Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:38.3129175Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:38.3232719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:38.3233593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:38.3233950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:38.3234283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:38.6076366Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T16:18:38.6130175Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13991 2022-09-27T16:18:38.6163242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13992 2022-09-27T16:18:38.6191166Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13993 2022-09-27T16:18:38.6219787Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13994 2022-09-27T16:18:39.7660958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:39.7661574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:39.7665079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:39.7665654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:39.7750142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:39.7750823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:39.7754908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:39.7755503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:39.7974380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:39.7975000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:39.7978189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:39.7978797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:39.8015021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:39.8015585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:39.8020379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:39.8021001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:39.8944939Z dist init r=2, world=4 2022-09-27T16:18:39.9021383Z dist init r=1, world=4 2022-09-27T16:18:39.9254602Z dist init r=3, world=4 2022-09-27T16:18:39.9289641Z dist init r=0, world=4 2022-09-27T16:18:39.9464246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:39.9565191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:39.9666952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:39.9667907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:39.9670030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:39.9671209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:39.9671900Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:39.9676445Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:39.9677435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:39.9678401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:39.9679097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:39.9679649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:40.3262134Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T16:18:40.3315631Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14047 2022-09-27T16:18:40.3351221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14048 2022-09-27T16:18:40.3380477Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14049 2022-09-27T16:18:40.3410351Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14050 2022-09-27T16:18:41.4871335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:41.4871866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:41.4874235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:41.4874938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:41.4975643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:41.4976298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:41.4980079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:41.4980704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:41.5240203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:41.5240856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:41.5241604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:41.5242120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:41.5244176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:41.5244799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:41.5245587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:41.5246167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:41.6153694Z dist init r=3, world=4 2022-09-27T16:18:41.6260319Z dist init r=1, world=4 2022-09-27T16:18:41.6510312Z dist init r=0, world=4 2022-09-27T16:18:41.6518434Z dist init r=2, world=4 2022-09-27T16:18:41.6730470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:41.6831750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:41.6832305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:41.6833575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:41.6834292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:41.6835352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:41.6836080Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:41.6836686Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:41.6842442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:41.6842990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:41.6843517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:41.6844029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:42.0452717Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:18:42.0506068Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14103 2022-09-27T16:18:42.0542755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14104 2022-09-27T16:18:42.0572480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14105 2022-09-27T16:18:42.0603198Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14106 2022-09-27T16:18:43.1855981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:43.1856723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:43.1859558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:43.1860262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:43.2063597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:43.2064249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:43.2068587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:43.2069218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:43.2140756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:43.2141361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:43.2145329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:43.2145967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:43.2359290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:43.2359910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:43.2362908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:43.2363516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:43.3143488Z dist init r=2, world=4 2022-09-27T16:18:43.3345145Z dist init r=0, world=4 2022-09-27T16:18:43.3407798Z dist init r=3, world=4 2022-09-27T16:18:43.3633363Z dist init r=1, world=4 2022-09-27T16:18:43.3818352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:43.3919139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:43.3957644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:43.3958141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:43.3958758Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:43.3959285Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:43.4022057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:43.4022618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:43.4065342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:43.4066072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:43.4066613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:43.4067127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:43.7644448Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:18:43.7697713Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14159 2022-09-27T16:18:43.7731957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14160 2022-09-27T16:18:43.7761113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14161 2022-09-27T16:18:43.7789454Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14162 2022-09-27T16:18:44.9235078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:44.9235744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:44.9239330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:44.9240067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:44.9468469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:44.9469052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:44.9471417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:44.9472003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:44.9787005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:44.9787617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:44.9790806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:44.9791398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:44.9792162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:44.9792771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:44.9796667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:44.9797563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:45.0673982Z dist init r=3, world=4 2022-09-27T16:18:45.0752654Z dist init r=1, world=4 2022-09-27T16:18:45.1084252Z dist init r=2, world=4 2022-09-27T16:18:45.1093988Z dist init r=0, world=4 2022-09-27T16:18:45.1293712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:45.1394970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:45.1496081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:45.1497023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:45.1497963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:45.1498518Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:45.1499184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:45.1499735Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:45.1603040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:45.1603585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:45.1604104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:45.1604651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:45.4830448Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T16:18:45.4883825Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14215 2022-09-27T16:18:45.4917200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14216 2022-09-27T16:18:45.4945734Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14217 2022-09-27T16:18:45.4974610Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14218 2022-09-27T16:18:46.6414595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:46.6415219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:46.6418639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:46.6419290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:46.6536569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:46.6537267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:46.6540890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:46.6541532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:46.6707078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:46.6707642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:46.6711792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:46.6712415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:46.6750949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:46.6751601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:46.6756476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:46.6757113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:46.7703304Z dist init r=2, world=4 2022-09-27T16:18:46.7818648Z dist init r=0, world=4 2022-09-27T16:18:46.7991337Z dist init r=1, world=4 2022-09-27T16:18:46.8025880Z dist init r=3, world=4 2022-09-27T16:18:46.8303284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:46.8303909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:46.8404168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:46.8405303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:46.8405710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:46.8406214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:46.8406730Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:46.8408842Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:46.8510985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:46.8511599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:46.8512138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:46.8512661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:47.2016161Z skip: Need at least 2 CUDA devices (1.718s) 2022-09-27T16:18:47.2069892Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14271 2022-09-27T16:18:47.2105476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14272 2022-09-27T16:18:47.2134065Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14273 2022-09-27T16:18:47.2163927Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14274 2022-09-27T16:18:48.3484005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:48.3484746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:48.3487784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:48.3488490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:48.3678061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:48.3678693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:48.3682446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:48.3682983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:48.3683727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:48.3684345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:48.3687714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:48.3688441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:48.3701326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:48.3702328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:48.3706276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:48.3706900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:48.4773596Z dist init r=1, world=4 2022-09-27T16:18:48.4977376Z dist init r=2, world=4 2022-09-27T16:18:48.4977723Z dist init r=3, world=4 2022-09-27T16:18:48.4984972Z dist init r=0, world=4 2022-09-27T16:18:48.5288518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:18:48.5288964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:18:48.5389012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:18:48.5389580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:18:48.5390248Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:48.5391109Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:48.5391788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:48.5392465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:18:48.5497431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:48.5498096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:18:48.5498620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:48.5499148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:18:48.8204014Z skip: Need at least 2 CUDA devices (1.619s) 2022-09-27T16:18:48.8204297Z 2022-09-27T16:18:48.8204721Z ---------------------------------------------------------------------- 2022-09-27T16:18:48.8205078Z Ran 8 tests in 13.997s 2022-09-27T16:18:48.8205253Z 2022-09-27T16:18:48.8205374Z OK (skipped=8) 2022-09-27T16:18:48.8205791Z 2022-09-27T16:18:48.8205920Z Generating XML reports... 2022-09-27T16:18:48.8247089Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20220927161834.xml 2022-09-27T16:18:49.1405339Z Running distributed/test_c10d_common ... [2022-09-27 16:18:49.140108] 2022-09-27T16:18:49.1405943Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:49.140195] 2022-09-27T16:18:50.3652983Z test_debug_level (__main__.CommTest) 2022-09-27T16:18:50.3653517Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) 2022-09-27T16:18:50.3654087Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) 2022-09-27T16:18:50.3654424Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) 2022-09-27T16:18:50.3654745Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) 2022-09-27T16:18:50.3655061Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) 2022-09-27T16:18:50.3655378Z test_collectives (__main__.PythonProcessGroupExtensionTest) 2022-09-27T16:18:50.3655931Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest) 2022-09-27T16:18:50.3656229Z test_send_recv (__main__.PythonProcessGroupExtensionTest) 2022-09-27T16:18:51.4410606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:51.4411040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:51.4412469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:51.4412823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:51.5767568Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-09-27T16:18:51.5777810Z 2022-09-27T16:18:51.5778259Z Running tests... 2022-09-27T16:18:51.5778655Z ---------------------------------------------------------------------- 2022-09-27T16:18:51.8186577Z test_debug_level (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14348 2022-09-27T16:18:51.8212582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14349 2022-09-27T16:18:52.9328253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:52.9328805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:52.9329790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:52.9330168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:52.9515853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:52.9516227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:52.9518394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:52.9518787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:53.0673416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:18:53.0858379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:18:53.3250795Z ok (1.747s) 2022-09-27T16:18:53.3251061Z 2022-09-27T16:18:53.3251572Z ---------------------------------------------------------------------- 2022-09-27T16:18:53.3251932Z Ran 1 test in 1.747s 2022-09-27T16:18:53.3252048Z 2022-09-27T16:18:53.3252107Z OK 2022-09-27T16:18:53.3252187Z 2022-09-27T16:18:53.3252281Z Generating XML reports... 2022-09-27T16:18:53.3299427Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220927161851.xml 2022-09-27T16:18:54.7333562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:54.7333959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:54.7335565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:54.7335960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:54.8690565Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-09-27T16:18:54.8700337Z 2022-09-27T16:18:54.8700688Z Running tests... 2022-09-27T16:18:54.8701076Z ---------------------------------------------------------------------- 2022-09-27T16:18:55.1108035Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.241s) 2022-09-27T16:18:55.1108294Z 2022-09-27T16:18:55.1108608Z ---------------------------------------------------------------------- 2022-09-27T16:18:55.1108850Z Ran 1 test in 0.241s 2022-09-27T16:18:55.1108966Z 2022-09-27T16:18:55.1109013Z OK 2022-09-27T16:18:55.1109105Z 2022-09-27T16:18:55.1109380Z Generating XML reports... 2022-09-27T16:18:55.1138721Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927161854.xml 2022-09-27T16:18:56.4832250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:56.4832631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:56.4834650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:56.4835008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:56.6177759Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-09-27T16:18:56.6186997Z 2022-09-27T16:18:56.6187239Z Running tests... 2022-09-27T16:18:56.6187898Z ---------------------------------------------------------------------- 2022-09-27T16:18:56.8550710Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.236s) 2022-09-27T16:18:56.8551041Z 2022-09-27T16:18:56.8551334Z ---------------------------------------------------------------------- 2022-09-27T16:18:56.8551597Z Ran 1 test in 0.236s 2022-09-27T16:18:56.8551713Z 2022-09-27T16:18:56.8551761Z OK 2022-09-27T16:18:56.8551852Z 2022-09-27T16:18:56.8551937Z Generating XML reports... 2022-09-27T16:18:56.8579338Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927161856.xml 2022-09-27T16:18:58.2144800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:58.2145195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:58.2147085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:58.2147453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:18:58.3510018Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-09-27T16:18:58.3519634Z 2022-09-27T16:18:58.3519725Z Running tests... 2022-09-27T16:18:58.3520284Z ---------------------------------------------------------------------- 2022-09-27T16:18:58.5904258Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.238s) 2022-09-27T16:18:58.5904495Z 2022-09-27T16:18:58.5904799Z ---------------------------------------------------------------------- 2022-09-27T16:18:58.5905047Z Ran 1 test in 0.238s 2022-09-27T16:18:58.5905165Z 2022-09-27T16:18:58.5905226Z OK 2022-09-27T16:18:58.5905320Z 2022-09-27T16:18:58.5905405Z Generating XML reports... 2022-09-27T16:18:58.5934472Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927161858.xml 2022-09-27T16:18:59.9568880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:18:59.9569523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:18:59.9570883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:18:59.9571472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:00.0927387Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-09-27T16:19:00.0937259Z 2022-09-27T16:19:00.0937681Z Running tests... 2022-09-27T16:19:00.0938314Z ---------------------------------------------------------------------- 2022-09-27T16:19:00.3295099Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.235s) 2022-09-27T16:19:00.3295511Z 2022-09-27T16:19:00.3295994Z ---------------------------------------------------------------------- 2022-09-27T16:19:00.3296426Z Ran 1 test in 0.236s 2022-09-27T16:19:00.3296630Z 2022-09-27T16:19:00.3296961Z OK 2022-09-27T16:19:00.3297118Z 2022-09-27T16:19:00.3297238Z Generating XML reports... 2022-09-27T16:19:00.3325642Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927161900.xml 2022-09-27T16:19:01.6877385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:01.6877749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:01.6878992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:01.6879349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:01.8243706Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-09-27T16:19:01.8252676Z 2022-09-27T16:19:01.8252962Z Running tests... 2022-09-27T16:19:01.8253319Z ---------------------------------------------------------------------- 2022-09-27T16:19:02.0786637Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14421 2022-09-27T16:19:02.0813492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14422 2022-09-27T16:19:02.0840178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14423 2022-09-27T16:19:02.0869627Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14424 2022-09-27T16:19:03.2371486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:03.2372109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:03.2373411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:03.2373890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:03.2535807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:03.2536456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:03.2537890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:03.2538509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:03.2646130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:03.2646755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:03.2648357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:03.2648927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:03.2692496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:03.2693088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:03.2696067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:03.2696698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:03.3739914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:19:03.3898472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:19:03.4004177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:03.4053482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:03.6912678Z ok (1.866s) 2022-09-27T16:19:03.6912924Z 2022-09-27T16:19:03.6913758Z ---------------------------------------------------------------------- 2022-09-27T16:19:03.6914025Z Ran 1 test in 1.866s 2022-09-27T16:19:03.6914214Z 2022-09-27T16:19:03.6914271Z OK 2022-09-27T16:19:03.6914364Z 2022-09-27T16:19:03.6914455Z Generating XML reports... 2022-09-27T16:19:03.6960967Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927161901.xml 2022-09-27T16:19:05.1238691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:05.1239118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:05.1240645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:05.1241031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:05.2580007Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-09-27T16:19:05.2588340Z 2022-09-27T16:19:05.2588535Z Running tests... 2022-09-27T16:19:05.2588986Z ---------------------------------------------------------------------- 2022-09-27T16:19:05.5029798Z test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14476 2022-09-27T16:19:05.5056445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14477 2022-09-27T16:19:05.5083481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14478 2022-09-27T16:19:05.5111053Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14479 2022-09-27T16:19:06.6551150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:06.6551771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:06.6553837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:06.6554447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:06.6596988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:06.6597578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:06.6600233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:06.6600827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:06.6691865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:06.6692515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:06.6695093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:06.6695703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:06.7233019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:06.7233707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:06.7235247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:06.7235792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:06.7908191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:19:06.7912607Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-09-27T16:19:06.7942898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:19:06.7948023Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-09-27T16:19:06.8037826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:06.8568595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:06.8576303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:19:07.7921267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:19:07.7954484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:19:07.7968550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:19:07.7969352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:07.7972646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:07.8023158Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:07.8057476Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:08.1170703Z ok (2.858s) 2022-09-27T16:19:08.1170874Z 2022-09-27T16:19:08.1171202Z ---------------------------------------------------------------------- 2022-09-27T16:19:08.1171504Z Ran 1 test in 2.858s 2022-09-27T16:19:08.1171619Z 2022-09-27T16:19:08.1171687Z OK 2022-09-27T16:19:08.1171832Z 2022-09-27T16:19:08.1171928Z Generating XML reports... 2022-09-27T16:19:08.1217839Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927161905.xml 2022-09-27T16:19:09.5058295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:09.5060390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:09.5061168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:09.5061691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:09.6412806Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-09-27T16:19:09.6423173Z 2022-09-27T16:19:09.6423584Z Running tests... 2022-09-27T16:19:09.6424010Z ---------------------------------------------------------------------- 2022-09-27T16:19:09.8954308Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14536 2022-09-27T16:19:09.8981427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14537 2022-09-27T16:19:09.9006353Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14538 2022-09-27T16:19:09.9033770Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14539 2022-09-27T16:19:11.0695533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:11.0696243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:11.0697279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:11.0697981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:11.0760521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:11.0761177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:11.0764758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:11.0765385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:11.0790802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:11.0791396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:11.0794446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:11.0795243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:11.0893991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:11.0894595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:11.0897049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:11.0897670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:11.2058113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:11.2120572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:19:11.2141415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:11.2249016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:19:11.5078691Z ok (1.865s) 2022-09-27T16:19:11.5078952Z 2022-09-27T16:19:11.5079296Z ---------------------------------------------------------------------- 2022-09-27T16:19:11.5079532Z Ran 1 test in 1.866s 2022-09-27T16:19:11.5079646Z 2022-09-27T16:19:11.5079708Z OK 2022-09-27T16:19:11.5079798Z 2022-09-27T16:19:11.5079890Z Generating XML reports... 2022-09-27T16:19:11.5125900Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927161909.xml 2022-09-27T16:19:12.9265617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:12.9266030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:12.9268195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:12.9268739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:13.0617688Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-09-27T16:19:13.0627111Z 2022-09-27T16:19:13.0627225Z Running tests... 2022-09-27T16:19:13.0627951Z ---------------------------------------------------------------------- 2022-09-27T16:19:13.3040135Z test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14591 2022-09-27T16:19:13.3066512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14592 2022-09-27T16:19:13.3092662Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14593 2022-09-27T16:19:13.3124127Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14594 2022-09-27T16:19:14.4449139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:14.4449750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:14.4451832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:14.4452427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:14.4610569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:14.4611418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:14.4612360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:14.4612993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:14.4917472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:14.4918057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:14.4919224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:14.4919773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:14.4923317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:14.4923918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:14.4927350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:14.4927948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:14.5809263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:14.5978216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:19:14.5985946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:19:14.6284731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:14.6285336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:19:14.6292802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:19:14.6293197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:19:14.6324065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:19:14.6325899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:14.6393182Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:14.6395804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:14.6396455Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:15.0170251Z ok (1.954s) 2022-09-27T16:19:15.0170500Z 2022-09-27T16:19:15.0171048Z ---------------------------------------------------------------------- 2022-09-27T16:19:15.0171347Z Ran 1 test in 1.954s 2022-09-27T16:19:15.0171454Z 2022-09-27T16:19:15.0171516Z OK 2022-09-27T16:19:15.0171614Z 2022-09-27T16:19:15.0171706Z Generating XML reports... 2022-09-27T16:19:15.0218485Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927161913.xml 2022-09-27T16:19:15.6041909Z Running distributed/elastic/timer/local_timer_example ... [2022-09-27 16:19:15.603806] 2022-09-27T16:19:15.6042555Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:15.603888] 2022-09-27T16:19:16.8217884Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example 2022-09-27T16:19:16.8228683Z 2022-09-27T16:19:16.8229337Z Running tests... 2022-09-27T16:19:16.8229747Z ---------------------------------------------------------------------- 2022-09-27T16:19:17.0682652Z test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2022-09-27 16:19:17,067 api: Starting LocalTimerServer... max_interval=0.01, daemon=True 2022-09-27T16:19:17.0683646Z [INFO] 2022-09-27 16:19:17,067 api: Starting watchdog thread... 2022-09-27T16:19:18.8346424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:18.8348286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:18.8359364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:18.8360924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:18.8549607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:18.8553631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:18.8555656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:18.8557161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:18.8736893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:18.8741137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:18.8744475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:18.8746602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:18.8919045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:18.8921425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:18.8927425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:18.8928210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:18.8929162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:18.8929889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:18.8935724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:18.8936451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:18.9066314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:18.9067000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:18.9072445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:18.9073227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:18.9207641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:18.9208268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:18.9211891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:18.9212519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:18.9220651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:18.9221282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:18.9224598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:18.9225504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:19.0538637Z [INFO] 2022-09-27 16:19:19,053 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:19.0758034Z [INFO] 2022-09-27 16:19:19,074 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:19.0758734Z [INFO] 2022-09-27 16:19:19,074 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:19.0946975Z [INFO] 2022-09-27 16:19:19,094 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:19.0988012Z [INFO] 2022-09-27 16:19:19,098 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:19.1258474Z [INFO] 2022-09-27 16:19:19,125 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:19.1405091Z [INFO] 2022-09-27 16:19:19,139 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:19.1588404Z [INFO] 2022-09-27 16:19:19,158 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:20.1781121Z [INFO] 2022-09-27 16:19:20,177 api: Reaping worker_id=[14658]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-09-27T16:19:20.1792504Z [INFO] 2022-09-27 16:19:20,177 api: Successfully reaped worker=[14658] 2022-09-27T16:19:20.2085441Z [INFO] 2022-09-27 16:19:20,208 api: Reaping worker_id=[14654]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-09-27T16:19:20.2085848Z [INFO] 2022-09-27 16:19:20,208 api: Successfully reaped worker=[14654] 2022-09-27T16:19:20.2390741Z [INFO] 2022-09-27 16:19:20,238 api: Reaping worker_id=[14652]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-09-27T16:19:20.2391367Z [INFO] 2022-09-27 16:19:20,238 api: Successfully reaped worker=[14652] 2022-09-27T16:19:20.2492329Z [INFO] 2022-09-27 16:19:20,248 api: Reaping worker_id=[14656]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-09-27T16:19:20.2492953Z [INFO] 2022-09-27 16:19:20,249 api: Successfully reaped worker=[14656] 2022-09-27T16:19:20.2555268Z [INFO] 2022-09-27 16:19:20,255 api: Stopping LocalTimerServer 2022-09-27T16:19:20.2555638Z [INFO] 2022-09-27 16:19:20,255 api: Stopping watchdog thread... 2022-09-27T16:19:20.2599645Z ok (3.437s) 2022-09-27T16:19:20.2619370Z test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2022-09-27 16:19:20,261 api: Starting LocalTimerServer... max_interval=0.01, daemon=True 2022-09-27T16:19:20.2619940Z [INFO] 2022-09-27 16:19:20,261 api: Starting watchdog thread... 2022-09-27T16:19:22.0770216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:22.0770760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:22.0786110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:22.0786864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:22.1021575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:22.1022156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:22.1022958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:22.1023566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:22.1082056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:22.1082764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:22.1086456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:22.1087139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:22.1149261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:22.1150123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:22.1152598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:22.1153358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:22.1255940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:22.1257033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:22.1258242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:22.1258839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:22.1264283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:22.1264939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:22.1265746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:22.1266365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:22.1339780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:22.1340542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:22.1346034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:22.1346687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:22.1686436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:22.1687038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:22.1690141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:22.1690802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:22.3101801Z [INFO] 2022-09-27 16:19:22,309 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:22.3128564Z [INFO] 2022-09-27 16:19:22,312 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:22.3160353Z [INFO] 2022-09-27 16:19:22,315 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:22.3220611Z [INFO] 2022-09-27 16:19:22,321 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:22.3398905Z [INFO] 2022-09-27 16:19:22,339 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:22.3464856Z [INFO] 2022-09-27 16:19:22,345 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:22.3666093Z [INFO] 2022-09-27 16:19:22,365 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:22.3750172Z [INFO] 2022-09-27 16:19:22,374 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:25.1973108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:25.1973768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:25.1987682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:25.1988188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:25.2418151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:25.2418797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:25.2436532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:25.2475987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:25.2635995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:25.2675547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:25.2676367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:25.2676837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:25.2677485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:25.2677966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:25.2678654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:25.2679126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:25.2876138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:25.2915917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:25.2916795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:25.2955809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:25.3124400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:25.3125077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:25.3128437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:25.3128964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:25.3138524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:25.3139170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:25.3139986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:25.3140509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:25.3144022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:25.3144681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:25.3149400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:25.3150052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:25.4238865Z [INFO] 2022-09-27 16:19:25,423 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:25.4435814Z [INFO] 2022-09-27 16:19:25,442 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:25.4715956Z [INFO] 2022-09-27 16:19:25,466 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:25.4755657Z [INFO] 2022-09-27 16:19:25,471 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:25.4955771Z [INFO] 2022-09-27 16:19:25,492 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:25.5210397Z [INFO] 2022-09-27 16:19:25,520 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:25.5223800Z [INFO] 2022-09-27 16:19:25,522 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:25.5503547Z [INFO] 2022-09-27 16:19:25,549 api: Timer client configured to: LocalTimerClient 2022-09-27T16:19:26.5555781Z [INFO] 2022-09-27 16:19:26,555 api: Reaping worker_id=[14830]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-09-27T16:19:26.5556263Z [INFO] 2022-09-27 16:19:26,555 api: Successfully reaped worker=[14830] 2022-09-27T16:19:26.5862423Z [INFO] 2022-09-27 16:19:26,585 api: Reaping worker_id=[14833]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-09-27T16:19:26.5862985Z [INFO] 2022-09-27 16:19:26,585 local_timer: Process with pid=14833 does not exist. Skipping 2022-09-27T16:19:26.5863343Z [INFO] 2022-09-27 16:19:26,585 api: Successfully reaped worker=[14833] 2022-09-27T16:19:26.5964811Z [INFO] 2022-09-27 16:19:26,596 api: Reaping worker_id=[14832]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-09-27T16:19:26.5965489Z [INFO] 2022-09-27 16:19:26,596 local_timer: Process with pid=14832 does not exist. Skipping 2022-09-27T16:19:26.5966091Z [INFO] 2022-09-27 16:19:26,596 api: Successfully reaped worker=[14832] 2022-09-27T16:19:26.5966511Z [INFO] 2022-09-27 16:19:26,596 api: Reaping worker_id=[14831]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-09-27T16:19:26.5966917Z [INFO] 2022-09-27 16:19:26,596 local_timer: Process with pid=14831 does not exist. Skipping 2022-09-27T16:19:26.5967268Z [INFO] 2022-09-27 16:19:26,596 api: Successfully reaped worker=[14831] 2022-09-27T16:19:26.6044198Z [INFO] 2022-09-27 16:19:26,604 api: Stopping LocalTimerServer 2022-09-27T16:19:26.6044671Z [INFO] 2022-09-27 16:19:26,604 api: Stopping watchdog thread... 2022-09-27T16:19:26.6070995Z ok (6.347s) 2022-09-27T16:19:26.6072442Z 2022-09-27T16:19:26.6072820Z ---------------------------------------------------------------------- 2022-09-27T16:19:26.6073142Z Ran 2 tests in 9.784s 2022-09-27T16:19:26.6073258Z 2022-09-27T16:19:26.6073319Z OK 2022-09-27T16:19:26.6073411Z 2022-09-27T16:19:26.6073490Z Generating XML reports... 2022-09-27T16:19:26.6105390Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220927161916.xml 2022-09-27T16:19:26.9383451Z Running distributed/_shard/checkpoint/test_file_system_checkpoint_cpu ... [2022-09-27 16:19:26.937937] 2022-09-27T16:19:26.9384099Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_file_system_checkpoint_cpu.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:26.938021] 2022-09-27T16:19:28.1598522Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu 2022-09-27T16:19:28.1611386Z 2022-09-27T16:19:28.1611466Z Running tests... 2022-09-27T16:19:28.1612316Z ---------------------------------------------------------------------- 2022-09-27T16:19:28.4060399Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14928 2022-09-27T16:19:28.4090139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14929 2022-09-27T16:19:29.5180352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:29.5180926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:29.5183049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:29.5183613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:29.5325624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:29.5326000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:29.5328987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:29.5329508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:29.6548265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:29.6650592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:29.6760535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:19:29.6761021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:19:29.6761790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:19:29.6762317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:19:30.0127918Z ok (1.851s) 2022-09-27T16:19:30.0197680Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14956 2022-09-27T16:19:30.0234194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14957 2022-09-27T16:19:31.1071229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:31.1071635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:31.1073181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:31.1073775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:31.1153617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:31.1154123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:31.1156887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:31.1157231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:31.2386442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:31.2469872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:31.2595744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:19:31.2596539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:19:31.2597209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:19:31.2597744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:19:31.7268721Z ok (1.714s) 2022-09-27T16:19:31.7322969Z test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14984 2022-09-27T16:19:31.7356846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14985 2022-09-27T16:19:32.8270571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:32.8271697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:32.8272204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:32.8272560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:32.8281365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:32.8281710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:32.8285034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:32.8285391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:32.9582891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:32.9600830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:32.9793047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:19:32.9793831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:19:32.9794962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:19:32.9795630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:19:33.3392016Z ok (1.612s) 2022-09-27T16:19:33.3457466Z test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15012 2022-09-27T16:19:33.3492297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15013 2022-09-27T16:19:34.4429454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:34.4429898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:34.4431506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:34.4431865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:34.4471602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:34.4471948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:34.4475263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:34.4475617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:34.5745533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:34.5794789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:34.5903052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:19:34.5903673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:19:34.5904300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:19:34.5904833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:19:35.1527313Z ok (1.813s) 2022-09-27T16:19:35.1726244Z test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.020s) 2022-09-27T16:19:35.1777397Z test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15040 2022-09-27T16:19:35.1811298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15041 2022-09-27T16:19:36.2579853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:36.2580229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:36.2582766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:36.2583127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:36.2641192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:36.2641535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:36.2645203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:36.2645564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:36.3902267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:36.3972634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:36.4183188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:19:36.4183929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:19:36.4184530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:19:36.4185056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:19:36.7843480Z ok (1.612s) 2022-09-27T16:19:36.7843738Z 2022-09-27T16:19:36.7844254Z ---------------------------------------------------------------------- 2022-09-27T16:19:36.7844662Z Ran 6 tests in 8.623s 2022-09-27T16:19:36.7844800Z 2022-09-27T16:19:36.7844863Z OK 2022-09-27T16:19:36.7844956Z 2022-09-27T16:19:36.7845048Z Generating XML reports... 2022-09-27T16:19:36.7879650Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20220927161928.xml 2022-09-27T16:19:36.7882033Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20220927161928.xml 2022-09-27T16:19:36.7884322Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20220927161928.xml 2022-09-27T16:19:37.1085926Z Running distributed/fsdp/test_fsdp_param_exec_order_wrap ... [2022-09-27 16:19:37.108230] 2022-09-27T16:19:37.1086518Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_param_exec_order_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:37.108319] 2022-09-27T16:19:38.3379384Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap 2022-09-27T16:19:38.3390865Z 2022-09-27T16:19:38.3390969Z Running tests... 2022-09-27T16:19:38.3392214Z ---------------------------------------------------------------------- 2022-09-27T16:19:38.3402595Z test_fsdp_flatten_params_exec_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) 2022-09-27T16:19:38.5824341Z Test ``_fsdp_params_exec_order`` with ``ParamExecOrderWrapPolicy``, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15079 2022-09-27T16:19:38.5850739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15080 2022-09-27T16:19:38.5877815Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15081 2022-09-27T16:19:38.5907632Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15082 2022-09-27T16:19:39.7376442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:39.7377035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:39.7380536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:39.7381200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:39.7414191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:39.7414784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:39.7420594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:39.7421461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:39.7425083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:39.7425895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:39.7432365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:39.7433025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:39.7457298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:39.7457880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:39.7458664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:39.7459263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:39.8656744Z dist init r=3, world=4 2022-09-27T16:19:39.8719047Z dist init r=2, world=4 2022-09-27T16:19:39.8769254Z dist init r=0, world=4 2022-09-27T16:19:39.8875434Z dist init r=1, world=4 2022-09-27T16:19:39.9067003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:19:39.9167804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:19:39.9270516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:19:39.9271230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:19:39.9272039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:39.9272747Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:39.9273271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:39.9273805Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:39.9377636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:39.9378453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:19:39.9378874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:39.9379288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:19:40.2956642Z skip: Need at least 2 CUDA devices (1.956s) 2022-09-27T16:19:40.2969117Z test_fsdp_flatten_params_exec_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) 2022-09-27T16:19:40.3014699Z Test ``_fsdp_params_exec_order`` with ``ParamExecOrderWrapPolicy``, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15135 2022-09-27T16:19:40.3048849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15136 2022-09-27T16:19:40.3076353Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15137 2022-09-27T16:19:40.3105149Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15138 2022-09-27T16:19:41.4310226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:41.4310974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:41.4314143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:41.4314882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:41.4679808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:41.4680569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:41.4684165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:41.4684798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:41.4685550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:41.4686067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:41.4689955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:41.4690602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:41.4844156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:41.4844762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:41.4848341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:41.4848986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:41.5600054Z dist init r=2, world=4 2022-09-27T16:19:41.5969835Z dist init r=1, world=4 2022-09-27T16:19:41.5974694Z dist init r=3, world=4 2022-09-27T16:19:41.6114707Z dist init r=0, world=4 2022-09-27T16:19:41.6285170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:19:41.6490447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:19:41.6490944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:19:41.6491321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:19:41.6491944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:41.6492475Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:41.6492995Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:41.6590268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:41.6598925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:19:41.6599460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:41.6599980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:19:41.6600392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:42.0147841Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:19:42.0157848Z test_fsdp_flatten_params_exec_order_symbolic_trace_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) 2022-09-27T16:19:42.0203831Z Tests ``ParamExecOrderWrapPolicy`` with symbolic tracing. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15191 2022-09-27T16:19:42.0236841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15192 2022-09-27T16:19:42.0265409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15193 2022-09-27T16:19:42.0300681Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15194 2022-09-27T16:19:43.1541230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:43.1541915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:43.1545127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:43.1545753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:43.1747948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:43.1748578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:43.1751426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:43.1751998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:43.1868895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:43.1869531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:43.1873412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:43.1874046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:43.1936329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:43.1936968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:43.1940785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:43.1941423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:43.2818516Z dist init r=1, world=4 2022-09-27T16:19:43.3031614Z dist init r=3, world=4 2022-09-27T16:19:43.3138108Z dist init r=2, world=4 2022-09-27T16:19:43.3210188Z dist init r=0, world=4 2022-09-27T16:19:43.3530603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:19:43.3531026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:19:43.3632283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:19:43.3633843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:19:43.3634909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:43.3635690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:43.3636497Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:43.3637294Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:43.3640272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:43.3640926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:19:43.3641443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:43.3641960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:19:43.7342349Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:19:43.7351589Z test_fsdp_flatten_params_exec_order_symbolic_trace_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) 2022-09-27T16:19:43.7398218Z Tests ``ParamExecOrderWrapPolicy`` with symbolic tracing. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15247 2022-09-27T16:19:43.7432959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15248 2022-09-27T16:19:43.7460872Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15249 2022-09-27T16:19:43.7489466Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15250 2022-09-27T16:19:44.9127603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:44.9128254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:44.9131495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:44.9132120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:44.9204586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:44.9205204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:44.9209244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:44.9209884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:44.9298045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:44.9298639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:44.9302869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:44.9303470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:44.9569064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:44.9569684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:44.9572594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:44.9573211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:45.0408558Z dist init r=2, world=4 2022-09-27T16:19:45.0490700Z dist init r=1, world=4 2022-09-27T16:19:45.0727438Z dist init r=3, world=4 2022-09-27T16:19:45.0836462Z dist init r=0, world=4 2022-09-27T16:19:45.1036107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:19:45.1222484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:19:45.1223334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:19:45.1223702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:19:45.1224308Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:45.1224822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:45.1225361Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:45.1238434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:45.1329698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:19:45.1330243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:45.1330786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:19:45.1331319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:45.4531715Z skip: Need at least 2 CUDA devices (1.719s) 2022-09-27T16:19:45.4532062Z 2022-09-27T16:19:45.4532540Z ---------------------------------------------------------------------- 2022-09-27T16:19:45.4532877Z Ran 4 tests in 7.114s 2022-09-27T16:19:45.4532982Z 2022-09-27T16:19:45.4533051Z OK (skipped=4) 2022-09-27T16:19:45.4533158Z 2022-09-27T16:19:45.4533243Z Generating XML reports... 2022-09-27T16:19:45.4569268Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap/TEST-TestFSDPExecOrder-20220927161938.xml 2022-09-27T16:19:45.7697754Z Running distributed/_shard/sharded_tensor/test_sharded_tensor ... [2022-09-27 16:19:45.769379] 2022-09-27T16:19:45.7698438Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:45.769462] 2022-09-27T16:19:47.0002773Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor 2022-09-27T16:19:47.0026957Z 2022-09-27T16:19:47.0027270Z Running tests... 2022-09-27T16:19:47.0027814Z ---------------------------------------------------------------------- 2022-09-27T16:19:47.0036681Z test_empty (__main__.TestCreateTensorFromParams) ... skip: CUDA GPU is needed (0.001s) 2022-09-27T16:19:47.0043713Z test_local_tensor (__main__.TestLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0050364Z test_local_tensor_error (__main__.TestLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0057370Z test_collect_local_shard (__main__.TestModuleHookApi) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0067649Z test_reshard_output (__main__.TestModuleHookApi) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0072426Z test_create_shard_with_no_placement (__main__.TestShardMetadata) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:19:47.0079959Z test_shard_metadata_init (__main__.TestShardMetadata) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0090923Z test_shard_parameter (__main__.TestShardParameter) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0105473Z test_shard_parameter_errors (__main__.TestShardParameter) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0114469Z test_shard_tensor (__main__.TestShardTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0126922Z test_shard_tensor_errors (__main__.TestShardTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0133533Z test_cleanup (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0151241Z test_complete_world_size (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0166246Z test_create_sharded_tensor_like (__main__.TestShardedTensorChunked) 2022-09-27T16:19:47.0166962Z Test tensor like methods, i.e. torch.zeros_like(...), torch.full_like, etc. ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0175599Z test_create_sharded_tensor_with_full (__main__.TestShardedTensorChunked) 2022-09-27T16:19:47.0176222Z Test sharded_tensor.full(...) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0183521Z test_create_sharded_tensor_with_ones (__main__.TestShardedTensorChunked) 2022-09-27T16:19:47.0184113Z Test sharded_tensor.ones(...) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0197559Z test_create_sharded_tensor_with_rand (__main__.TestShardedTensorChunked) 2022-09-27T16:19:47.0198163Z Test sharded_tensor.rand(...)/randn(...) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0206145Z test_create_sharded_tensor_with_zeros (__main__.TestShardedTensorChunked) 2022-09-27T16:19:47.0206548Z Test sharded_tensor.zeros(...) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0213632Z test_gather_even (__main__.TestShardedTensorChunked) 2022-09-27T16:19:47.0214284Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0221644Z test_gather_uneven (__main__.TestShardedTensorChunked) 2022-09-27T16:19:47.0222054Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0232742Z test_insufficient_sharding_dims (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0239666Z test_invalid_pg_rpc_ranks (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0258121Z test_invalid_sharding (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0269119Z test_load_state_dict_errors (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0282767Z test_multiple_local_shards (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0298896Z test_new_group (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0314206Z test_partial_world_size (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0327847Z test_sharded_tensor_metadata (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0344685Z test_sharded_tensor_sizes (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0354807Z test_sharding_columns (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0365153Z test_state_dict (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0374109Z test_state_dict_new_group (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0381604Z test_state_dict_no_sharded_tensors (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0388430Z test_custom_op (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0393694Z test_custom_op_errors (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:19:47.0401930Z test_custom_op_override (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0412900Z test_create_sharded_tensor_with_ones (__main__.TestShardedTensorEnumerable) 2022-09-27T16:19:47.0413331Z Test sharded_tensor.ones(...) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0423840Z test_gather_even (__main__.TestShardedTensorEnumerable) 2022-09-27T16:19:47.0424327Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0434870Z test_gather_uneven (__main__.TestShardedTensorEnumerable) 2022-09-27T16:19:47.0435290Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0454501Z test_grid_sharding (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0475955Z test_multiple_local_shards (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0493904Z test_new_group (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0513436Z test_partial_world_size (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0519907Z test_sharded_tensor_device (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0538052Z test_sharded_tensor_metadata (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0559185Z test_sharded_tensor_to_cpu (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0573369Z test_sharded_tensor_to_cuda (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.0597523Z test_sharded_tensor_to_test (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0618081Z test_uneven_shards (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0637016Z test_with_rpc_names (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0654671Z test_init_from_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0677555Z test_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:47.0704576Z test_init_from_local_shards_and_global_metadata_invalid_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.003s) 2022-09-27T16:19:47.0717737Z test_init_from_local_shards_invalid_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:47.3217200Z test_init_from_local_shards_invalid_pin_memory (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15314 2022-09-27T16:19:47.3243793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15315 2022-09-27T16:19:47.3271092Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15316 2022-09-27T16:19:47.3301063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15317 2022-09-27T16:19:48.5826825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:48.5827438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:48.5831034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:48.5831671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:48.6281212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:48.6281871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:48.6284681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:48.6285331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:48.6487494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:48.6488145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:48.6491111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:48.6491759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:48.7111728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:19:48.7289904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:48.7290565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:48.7294257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:48.7294832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:48.7557816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:48.7755753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:19:48.8538077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:48.8851476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:19:48.8851897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:19:48.8951958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:19:48.8953081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:19:48.8954145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:48.8954820Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:48.8955355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:48.8955882Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:19:49.2351873Z skip: Need at least 4 CUDA devices (2.163s) 2022-09-27T16:19:49.2368102Z test_init_from_local_shards_invalid_property_cross_ranks (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-09-27T16:19:49.2375017Z test_init_from_local_shards_invalid_shards_gaps (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:49.2381184Z test_init_from_local_shards_invalid_shards_overlap (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:49.2395946Z test_init_from_local_shards_new_group (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:49.2405613Z test_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:49.2467371Z test_st_base_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15370 2022-09-27T16:19:49.2501851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15371 2022-09-27T16:19:49.2530315Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15372 2022-09-27T16:19:49.2559220Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15373 2022-09-27T16:19:50.3683376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:50.3683908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:50.3686266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:50.3686754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:50.3741675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:50.3742206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:50.3745491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:50.3745995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:50.3989777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:50.3990447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:50.3994329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:50.3995126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:50.4306005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:50.4306687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:50.4309726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:50.4310353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:50.4964837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:19:50.5011181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:50.5384675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:19:50.5575015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:50.8599900Z skip: Need at least 4 CUDA devices (1.619s) 2022-09-27T16:19:50.8607294Z test_init_from_local_tensor (__main__.TestShardedTensorFromLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:50.8616260Z test_init_from_local_tensor_errors (__main__.TestShardedTensorFromLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:19:50.8962833Z test_serialize_and_deserialize (__main__.TestShardedTensorMetadata) ... ok (0.034s) 2022-09-27T16:19:50.8963170Z 2022-09-27T16:19:50.8963597Z ---------------------------------------------------------------------- 2022-09-27T16:19:50.8963987Z Ran 64 tests in 3.894s 2022-09-27T16:19:50.8964183Z 2022-09-27T16:19:50.8964285Z OK (skipped=63) 2022-09-27T16:19:50.8964444Z 2022-09-27T16:19:50.8964577Z Generating XML reports... 2022-09-27T16:19:50.8996357Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20220927161946.xml 2022-09-27T16:19:50.8999434Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20220927161946.xml 2022-09-27T16:19:50.9004796Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20220927161946.xml 2022-09-27T16:19:50.9009689Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20220927161946.xml 2022-09-27T16:19:50.9013809Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20220927161946.xml 2022-09-27T16:19:50.9018403Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20220927161946.xml 2022-09-27T16:19:50.9023085Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20220927161946.xml 2022-09-27T16:19:50.9047509Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20220927161946.xml 2022-09-27T16:19:50.9053076Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20220927161946.xml 2022-09-27T16:19:50.9068895Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20220927161946.xml 2022-09-27T16:19:50.9081814Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20220927161946.xml 2022-09-27T16:19:50.9086244Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20220927161946.xml 2022-09-27T16:19:51.2302366Z Running distributed/elastic/timer/local_timer_test ... [2022-09-27 16:19:51.229810] 2022-09-27T16:19:51.2303223Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:51.229886] 2022-09-27T16:19:52.4369779Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_test 2022-09-27T16:19:52.4381855Z 2022-09-27T16:19:52.4382180Z Running tests... 2022-09-27T16:19:52.4382769Z ---------------------------------------------------------------------- 2022-09-27T16:19:52.4389879Z test_acquire_release (__main__.LocalTimerServerTest) 2022-09-27T16:19:52.6754441Z tests that: ... ok (0.237s) 2022-09-27T16:19:52.6759232Z test_expired_timers (__main__.LocalTimerServerTest) 2022-09-27T16:19:52.6773218Z tests that a single expired timer on a process should terminate ... ok (0.002s) 2022-09-27T16:19:52.6783311Z test_valid_timers (__main__.LocalTimerServerTest) 2022-09-27T16:19:52.6798685Z tests that valid timers are processed correctly and the process is left alone ... ok (0.002s) 2022-09-27T16:19:52.6805156Z test_watchdog_call_count (__main__.LocalTimerServerTest) 2022-09-27T16:19:52.7824052Z checks that the watchdog function ran wait/interval +- 1 times ... ok (0.102s) 2022-09-27T16:19:52.7825806Z test_watchdog_empty_queue (__main__.LocalTimerServerTest) 2022-09-27T16:19:52.7934196Z checks that the watchdog can run on an empty queue ... ok (0.011s) 2022-09-27T16:19:52.8003626Z test_client_interaction (__main__.LocalTimerTest) ... ok (0.007s) 2022-09-27T16:19:52.8119534Z test_exception_propagation (__main__.LocalTimerTest) ... ok (0.011s) 2022-09-27T16:19:52.8126595Z test_get_timer_recursive (__main__.LocalTimerTest) 2022-09-27T16:19:54.1289409Z If a function acquires a countdown timer with default scope, ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:54.1289870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:54.1627892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:54.1628266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:54.4659087Z ok (1.654s) 2022-09-27T16:19:54.5692491Z test_happy_path (__main__.LocalTimerTest) ... ok (0.103s) 2022-09-27T16:19:54.5806386Z test_no_client (__main__.LocalTimerTest) ... ok (0.011s) 2022-09-27T16:19:54.7091921Z test_timer (__main__.LocalTimerTest) ... ok (0.128s) 2022-09-27T16:19:54.7318958Z test_get (__main__.MultiprocessingRequestQueueTest) ... ok (0.022s) 2022-09-27T16:19:54.7325981Z test_get_less_than_size (__main__.MultiprocessingRequestQueueTest) 2022-09-27T16:19:55.2389577Z Tests slow producer. ... ok (0.507s) 2022-09-27T16:19:55.2401845Z test_get_size (__main__.MultiprocessingRequestQueueTest) 2022-09-27T16:19:56.1487968Z Creates a "producer" process that enqueues ``n`` elements ... ok (0.909s) 2022-09-27T16:19:56.1488322Z 2022-09-27T16:19:56.1488729Z ---------------------------------------------------------------------- 2022-09-27T16:19:56.1489126Z Ran 14 tests in 3.711s 2022-09-27T16:19:56.1489234Z 2022-09-27T16:19:56.1489294Z OK 2022-09-27T16:19:56.1489432Z 2022-09-27T16:19:56.1489541Z Generating XML reports... 2022-09-27T16:19:56.1535506Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20220927161952.xml 2022-09-27T16:19:56.1542711Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20220927161952.xml 2022-09-27T16:19:56.1547317Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20220927161952.xml 2022-09-27T16:19:56.5662103Z Running distributed/fsdp/test_distributed_checkpoint ... [2022-09-27 16:19:56.565777] 2022-09-27T16:19:56.5663012Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:56.565862] 2022-09-27T16:19:57.7928281Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint 2022-09-27T16:19:57.7939408Z 2022-09-27T16:19:57.7939852Z Running tests... 2022-09-27T16:19:57.7940256Z ---------------------------------------------------------------------- 2022-09-27T16:19:58.0383547Z test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15468 2022-09-27T16:19:58.0409896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15469 2022-09-27T16:19:59.1601211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:59.1601855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:59.1611708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:59.1612353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:59.1682284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:19:59.1682882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:19:59.1694212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:19:59.1694604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:19:59.2885644Z dist init r=0, world=2 2022-09-27T16:19:59.2956906Z dist init r=1, world=2 2022-09-27T16:19:59.3094554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:19:59.3095006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:19:59.3095635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:19:59.3096165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:19:59.3200705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:19:59.3201266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:19:59.6448825Z skip: Need at least 2 CUDA devices (1.851s) 2022-09-27T16:19:59.6509346Z test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15496 2022-09-27T16:19:59.6543222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15497 2022-09-27T16:20:00.7506232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:00.7506872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:00.7515554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:00.7515915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:00.7688829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:00.7689210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:00.7698792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:00.7699389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:00.8758779Z dist init r=0, world=2 2022-09-27T16:20:00.8946668Z dist init r=1, world=2 2022-09-27T16:20:00.9055160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:20:00.9055841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:20:00.9056701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:20:00.9057227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-09-27T16:20:00.9161608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:20:00.9162198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:20:01.2576983Z skip: Need at least 2 CUDA devices (1.613s) 2022-09-27T16:20:01.2577321Z 2022-09-27T16:20:01.2577810Z ---------------------------------------------------------------------- 2022-09-27T16:20:01.2578089Z Ran 2 tests in 3.464s 2022-09-27T16:20:01.2578204Z 2022-09-27T16:20:01.2578262Z OK (skipped=2) 2022-09-27T16:20:01.2578371Z 2022-09-27T16:20:01.2578454Z Generating XML reports... 2022-09-27T16:20:01.2613941Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20220927161957.xml 2022-09-27T16:20:01.6081627Z Running distributed/elastic/utils/distributed_test ... [2022-09-27 16:20:01.607775] 2022-09-27T16:20:01.6082222Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:01.607864] 2022-09-27T16:20:02.8939321Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test 2022-09-27T16:20:02.8951218Z 2022-09-27T16:20:02.8951526Z Running tests... 2022-09-27T16:20:02.8952180Z ---------------------------------------------------------------------- 2022-09-27T16:20:03.1414550Z test_create_store_multi (__main__.DistributedUtilTest) ... ok (0.246s) 2022-09-27T16:20:03.1426468Z test_create_store_no_port_multi (__main__.DistributedUtilTest) ... ok (0.001s) 2022-09-27T16:20:03.1432919Z test_create_store_single_server (__main__.DistributedUtilTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/66207 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.000s) 2022-09-27T16:20:06.1590983Z test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... ok (3.016s) 2022-09-27T16:20:06.1600363Z test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (51c56aedd00a, 0). 2022-09-27T16:20:06.1601336Z ok (0.001s) 2022-09-27T16:20:06.1615392Z test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:38939 (errno: 98 - Address already in use). 2022-09-27T16:20:06.1620042Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:38939 (errno: 98 - Address already in use). 2022-09-27T16:20:06.1620408Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address. 2022-09-27T16:20:06.1622764Z ok (0.002s) 2022-09-27T16:20:06.1649989Z test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (51c56aedd00a, 48727). 2022-09-27T16:20:06.1650820Z ok (0.003s) 2022-09-27T16:20:06.1652000Z 2022-09-27T16:20:06.1652400Z ---------------------------------------------------------------------- 2022-09-27T16:20:06.1652819Z Ran 7 tests in 3.270s 2022-09-27T16:20:06.1653003Z 2022-09-27T16:20:06.1653117Z OK (skipped=1) 2022-09-27T16:20:06.1653297Z 2022-09-27T16:20:06.1653446Z Generating XML reports... 2022-09-27T16:20:06.1687004Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220927162002.xml 2022-09-27T16:20:06.4839197Z Running distributed/fsdp/test_fsdp_pure_fp16 ... [2022-09-27 16:20:06.483537] 2022-09-27T16:20:06.4839809Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:06.483623] 2022-09-27T16:20:07.7198490Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16 2022-09-27T16:20:07.7209339Z 2022-09-27T16:20:07.7209482Z Running tests... 2022-09-27T16:20:07.7210326Z ---------------------------------------------------------------------- 2022-09-27T16:20:07.7216834Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-09-27T16:20:07.9543129Z Tests pure FP16 training, including when the parameter's dtype is ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/73315 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.233s) 2022-09-27T16:20:07.9546591Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=True) (__main__.TestPureFP16) 2022-09-27T16:20:07.9634897Z Tests pure FP16 training, including when the parameter's dtype is ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15555 2022-09-27T16:20:07.9661692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15556 2022-09-27T16:20:07.9688642Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15557 2022-09-27T16:20:07.9718954Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15558 2022-09-27T16:20:09.1227775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:09.1228455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:09.1231845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:09.1232516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:09.1291463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:09.1292086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:09.1295698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:09.1296347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:09.1445803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:09.1446597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:09.1451037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:09.1451661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:09.1530630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:09.1531217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:09.1535245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:09.1535907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:09.2521303Z dist init r=1, world=4 2022-09-27T16:20:09.2565705Z dist init r=2, world=4 2022-09-27T16:20:09.2720695Z dist init r=3, world=4 2022-09-27T16:20:09.2792996Z dist init r=0, world=4 2022-09-27T16:20:09.3206479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:20:09.3306836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:20:09.3409970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:20:09.3410564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:20:09.3411306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:20:09.3411921Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:20:09.3412443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:20:09.3412965Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:20:09.3417030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:20:09.3417601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:20:09.3418413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:20:09.3418937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:20:09.6767790Z skip: Need at least 2 CUDA devices (1.722s) 2022-09-27T16:20:09.6768007Z 2022-09-27T16:20:09.6768388Z ---------------------------------------------------------------------- 2022-09-27T16:20:09.6768631Z Ran 2 tests in 1.956s 2022-09-27T16:20:09.6768748Z 2022-09-27T16:20:09.6768829Z OK (skipped=2) 2022-09-27T16:20:09.6768939Z 2022-09-27T16:20:09.6769043Z Generating XML reports... 2022-09-27T16:20:09.6803268Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20220927162007.xml 2022-09-27T16:20:10.0241150Z Running distributed/fsdp/test_fsdp_fx ... [2022-09-27 16:20:10.023744] 2022-09-27T16:20:10.0241979Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:10.023827] 2022-09-27T16:20:11.2557916Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx 2022-09-27T16:20:11.2569116Z 2022-09-27T16:20:11.2569216Z Running tests... 2022-09-27T16:20:11.2569941Z ---------------------------------------------------------------------- 2022-09-27T16:20:11.2592753Z test_symbolic_tracing_outputs (__main__.TestSymbolicTracing) 2022-09-27T16:20:11.5059595Z test ``execution_info.module_forward_order`` and ``execution_info.module_to_execution_infos`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15622 2022-09-27T16:20:11.5087421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15623 2022-09-27T16:20:11.5114148Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15624 2022-09-27T16:20:11.5145237Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15625 2022-09-27T16:20:12.6822296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:12.6822984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:12.6825587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:12.6826185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:12.6924178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:12.6927155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:12.6927976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:12.6928572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:12.6938696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:12.6939338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:12.6943528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:12.6944100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:12.6953775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:12.6954352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:12.6958736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:12.6959386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:12.8261471Z dist init r=3, world=4 2022-09-27T16:20:12.8269315Z dist init r=0, world=4 2022-09-27T16:20:12.8269653Z dist init r=2, world=4 2022-09-27T16:20:12.8271914Z dist init r=1, world=4 2022-09-27T16:20:12.8584013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:20:12.8684750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:20:12.8789215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:20:12.8789873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:20:12.8790586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:20:12.8791276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:20:12.8792050Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:20:12.8792862Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:20:12.8797344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:20:12.8797922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:20:12.8798775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:20:12.8799429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:20:13.3193043Z ok (2.062s) 2022-09-27T16:20:13.3193297Z 2022-09-27T16:20:13.3193823Z ---------------------------------------------------------------------- 2022-09-27T16:20:13.3194111Z Ran 1 test in 2.062s 2022-09-27T16:20:13.3194232Z 2022-09-27T16:20:13.3194279Z OK 2022-09-27T16:20:13.3194372Z 2022-09-27T16:20:13.3194470Z Generating XML reports... 2022-09-27T16:20:13.3226267Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20220927162011.xml 2022-09-27T16:20:13.6547301Z Running distributed/fsdp/test_fsdp_uneven ... [2022-09-27 16:20:13.654323] 2022-09-27T16:20:13.6547860Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:13.654408] 2022-09-27T16:20:14.8749436Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven 2022-09-27T16:20:14.8760989Z 2022-09-27T16:20:14.8761155Z Running tests... 2022-09-27T16:20:14.8761783Z ---------------------------------------------------------------------- 2022-09-27T16:20:14.8773079Z test_one_iteration (__main__.TestUnevenParamShard) 2022-09-27T16:20:15.1198813Z Test FSDP with uneven divide of parameter shards. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15689 2022-09-27T16:20:15.1225471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15690 2022-09-27T16:20:15.1253171Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15691 2022-09-27T16:20:15.1281923Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15692 2022-09-27T16:20:16.2696450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:16.2697096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:16.2700201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:16.2701035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:16.2877466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:16.2878091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:16.2880928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:16.2881580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:16.2905622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:16.2906181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:16.2911441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:16.2912079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:16.3002470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:16.3003117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:16.3007712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:16.3008348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:16.3968671Z dist init r=2, world=4 2022-09-27T16:20:16.4150373Z dist init r=3, world=4 2022-09-27T16:20:16.4169736Z dist init r=0, world=4 2022-09-27T16:20:16.4265688Z dist init r=1, world=4 2022-09-27T16:20:16.4459626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:20:16.4561019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-09-27T16:20:16.4663161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-09-27T16:20:16.4664239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:20:16.4664870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-09-27T16:20:16.4665652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:20:16.4666460Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:20:16.4667348Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-09-27T16:20:16.4671947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:20:16.4672519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:20:16.4672874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:20:16.4673339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:20:16.8329158Z skip: Need at least 2 CUDA devices (1.956s) 2022-09-27T16:20:16.8329472Z 2022-09-27T16:20:16.8329977Z ---------------------------------------------------------------------- 2022-09-27T16:20:16.8330302Z Ran 1 test in 1.957s 2022-09-27T16:20:16.8330417Z 2022-09-27T16:20:16.8330489Z OK (skipped=1) 2022-09-27T16:20:16.8330585Z 2022-09-27T16:20:16.8330670Z Generating XML reports... 2022-09-27T16:20:16.8366234Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20220927162014.xml 2022-09-27T16:20:17.1752756Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-09-27 16:20:17.174849] 2022-09-27T16:20:17.1753400Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:17.174930] 2022-09-27T16:20:18.3852248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_b8asdc 2022-09-27T16:20:18.3853044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_b8asdc/_remote_module_non_scriptable.py 2022-09-27T16:20:18.3977443Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec 2022-09-27T16:20:18.3990822Z 2022-09-27T16:20:18.3990927Z Running tests... 2022-09-27T16:20:18.3991491Z ---------------------------------------------------------------------- 2022-09-27T16:20:18.6487926Z test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15756 2022-09-27T16:20:18.6517391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15757 2022-09-27T16:20:18.6545405Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15758 2022-09-27T16:20:18.6577246Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15759 2022-09-27T16:20:19.7915292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:19.7915921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:19.7927799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:19.7928444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:19.7987769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:19.7988460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:19.8295675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:19.8296611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:19.8310661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:19.8311332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:19.8365082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:19.8365706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:19.8391730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:19.8392354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:19.8749386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:19.8750076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:19.9597339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyob55qb 2022-09-27T16:20:19.9598184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyob55qb/_remote_module_non_scriptable.py 2022-09-27T16:20:19.9611628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxdhs68k5 2022-09-27T16:20:19.9613799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxdhs68k5/_remote_module_non_scriptable.py 2022-09-27T16:20:19.9711916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-09-27T16:20:19.9725773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-09-27T16:20:19.9831159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6sfohjxm 2022-09-27T16:20:19.9833387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6sfohjxm/_remote_module_non_scriptable.py 2022-09-27T16:20:19.9944245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-09-27T16:20:20.0082500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusjmqsny 2022-09-27T16:20:20.0084321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusjmqsny/_remote_module_non_scriptable.py 2022-09-27T16:20:20.0197072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:20:20.3621872Z ok (1.963s) 2022-09-27T16:20:20.3631017Z test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec) 2022-09-27T16:20:20.3631517Z Test custom spec can be invoked from the ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:20.3639363Z test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec) 2022-09-27T16:20:20.3640008Z Test sharded_tensor.ones(...) with the custom ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:20.3654388Z test_chunked_sharding_spec (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.001s) 2022-09-27T16:20:20.3662007Z test_device_placement (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.001s) 2022-09-27T16:20:20.3698993Z test_enumerable_sharding_spec (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.004s) 2022-09-27T16:20:20.3719923Z test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s) 2022-09-27T16:20:20.3730897Z test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s) 2022-09-27T16:20:20.3741811Z test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s) 2022-09-27T16:20:20.3817973Z test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.007s) 2022-09-27T16:20:20.3818289Z 2022-09-27T16:20:20.3818751Z ---------------------------------------------------------------------- 2022-09-27T16:20:20.3819274Z Ran 10 tests in 1.983s 2022-09-27T16:20:20.3819450Z 2022-09-27T16:20:20.3819562Z OK (skipped=5) 2022-09-27T16:20:20.3819728Z 2022-09-27T16:20:20.3819850Z Generating XML reports... 2022-09-27T16:20:20.3863812Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220927162018.xml 2022-09-27T16:20:20.3873076Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220927162018.xml 2022-09-27T16:20:20.6943266Z Running distributed/fsdp/test_fsdp_overlap ... [2022-09-27 16:20:20.693928] 2022-09-27T16:20:20.6943970Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:20.694013] 2022-09-27T16:20:21.9317402Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap 2022-09-27T16:20:21.9329390Z 2022-09-27T16:20:21.9329489Z Running tests... 2022-09-27T16:20:21.9330461Z ---------------------------------------------------------------------- 2022-09-27T16:20:22.1901881Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15811 2022-09-27T16:20:23.2779302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:23.2779705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:23.2781762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:23.2782121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:23.4025115Z dist init r=0, world=1 2022-09-27T16:20:23.4033516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-09-27T16:20:23.4034160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-09-27T16:20:23.4037783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-09-27T16:20:23.6933279Z skip: Need at least 2 CUDA devices (1.760s) 2022-09-27T16:20:23.6944998Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-09-27T16:20:23.6945699Z 2022-09-27T16:20:23.6945925Z ---------------------------------------------------------------------- 2022-09-27T16:20:23.6946175Z Ran 2 tests in 1.761s 2022-09-27T16:20:23.6946291Z 2022-09-27T16:20:23.6946362Z OK (skipped=2) 2022-09-27T16:20:23.6946468Z 2022-09-27T16:20:23.6946557Z Generating XML reports... 2022-09-27T16:20:23.6977284Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220927162021.xml 2022-09-27T16:20:23.6980030Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220927162021.xml 2022-09-27T16:20:24.0008855Z Running distributed/_shard/checkpoint/test_planner ... [2022-09-27 16:20:24.000473] 2022-09-27T16:20:24.0009688Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_planner.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:24.000553] 2022-09-27T16:20:25.2448739Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_planner 2022-09-27T16:20:25.2459812Z 2022-09-27T16:20:25.2460047Z Running tests... 2022-09-27T16:20:25.2460435Z ---------------------------------------------------------------------- 2022-09-27T16:20:25.4936915Z test_global_plan (__main__.TestSavePlan) ... ok (0.248s) 2022-09-27T16:20:25.4973710Z test_load_with_resharding (__main__.TestSavePlan) ... ok (0.004s) 2022-09-27T16:20:25.5001694Z test_load_with_world_size_diff_by_one (__main__.TestSavePlan) ... ok (0.003s) 2022-09-27T16:20:25.5032411Z test_local_load_plan (__main__.TestSavePlan) ... ok (0.003s) 2022-09-27T16:20:25.5062040Z test_local_plan (__main__.TestSavePlan) ... ok (0.003s) 2022-09-27T16:20:25.5062459Z 2022-09-27T16:20:25.5063261Z ---------------------------------------------------------------------- 2022-09-27T16:20:25.5063614Z Ran 5 tests in 0.260s 2022-09-27T16:20:25.5063726Z 2022-09-27T16:20:25.5063792Z OK 2022-09-27T16:20:25.5063886Z 2022-09-27T16:20:25.5063981Z Generating XML reports... 2022-09-27T16:20:25.5090693Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_planner/TEST-TestSavePlan-20220927162025.xml 2022-09-27T16:20:25.8000308Z Running distributed/_shard/checkpoint/test_utils ... [2022-09-27 16:20:25.799585] 2022-09-27T16:20:25.8000949Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:25.799665] 2022-09-27T16:20:27.0166276Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_utils 2022-09-27T16:20:27.0195584Z 2022-09-27T16:20:27.0196103Z Running tests... 2022-09-27T16:20:27.0196785Z ---------------------------------------------------------------------- 2022-09-27T16:20:27.2655243Z test_flat_data (__main__.TestMedatadaIndex) ... ok (0.247s) 2022-09-27T16:20:27.2663520Z test_index_hint_ignored_on_equals (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-09-27T16:20:27.2671048Z test_index_hint_ignored_on_hash (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-09-27T16:20:27.2679034Z test_init_convert_offset (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-09-27T16:20:27.2711153Z test_sharded_tensor_lookup (__main__.TestMedatadaIndex) ... ok (0.003s) 2022-09-27T16:20:27.2711473Z 2022-09-27T16:20:27.2712008Z ---------------------------------------------------------------------- 2022-09-27T16:20:27.2712322Z Ran 5 tests in 0.253s 2022-09-27T16:20:27.2712443Z 2022-09-27T16:20:27.2712508Z OK 2022-09-27T16:20:27.2712609Z 2022-09-27T16:20:27.2712750Z Generating XML reports... 2022-09-27T16:20:27.2739391Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_utils/TEST-TestMedatadaIndex-20220927162027.xml 2022-09-27T16:20:27.5657092Z Running distributed/test_launcher ... [2022-09-27 16:20:27.565319] 2022-09-27T16:20:27.5657674Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:27.565402] 2022-09-27T16:20:29.1248908Z Test results will be stored in test-reports/python-unittest/distributed.test_launcher 2022-09-27T16:20:29.1260212Z 2022-09-27T16:20:29.1260369Z Running tests... 2022-09-27T16:20:29.1260908Z ---------------------------------------------------------------------- 2022-09-27T16:20:29.3669836Z test_launch_user_script (__main__.TestDistributedLaunch) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79488 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.241s) 2022-09-27T16:20:29.3670322Z 2022-09-27T16:20:29.3670547Z ---------------------------------------------------------------------- 2022-09-27T16:20:29.3670843Z Ran 1 test in 0.241s 2022-09-27T16:20:29.3670961Z 2022-09-27T16:20:29.3671033Z OK (skipped=1) 2022-09-27T16:20:29.3671140Z 2022-09-27T16:20:29.3671223Z Generating XML reports... 2022-09-27T16:20:29.3693329Z Generated XML report: test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20220927162029.xml 2022-09-27T16:20:29.6686259Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2022-09-27 16:20:29.668224] 2022-09-27T16:20:29.6686872Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:29.668304] 2022-09-27T16:20:30.9090837Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops 2022-09-27T16:20:30.9101350Z 2022-09-27T16:20:30.9101456Z Running tests... 2022-09-27T16:20:30.9101919Z ---------------------------------------------------------------------- 2022-09-27T16:20:30.9111596Z test_sharded_dropout (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:30.9118835Z test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:30.9125131Z test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:30.9132362Z test_sharded_tensor_nan_to_num (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:30.9132679Z 2022-09-27T16:20:30.9133359Z ---------------------------------------------------------------------- 2022-09-27T16:20:30.9133643Z Ran 4 tests in 0.003s 2022-09-27T16:20:30.9133760Z 2022-09-27T16:20:30.9133844Z OK (skipped=4) 2022-09-27T16:20:30.9133951Z 2022-09-27T16:20:30.9134034Z Generating XML reports... 2022-09-27T16:20:30.9159510Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220927162030.xml 2022-09-27T16:20:31.1237191Z Running distributed/_shard/sharded_tensor/ops/test_init ... [2022-09-27 16:20:31.123361] 2022-09-27T16:20:31.1237791Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:31.123442] 2022-09-27T16:20:32.3367077Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init 2022-09-27T16:20:32.3378085Z 2022-09-27T16:20:32.3378187Z Running tests... 2022-09-27T16:20:32.3378646Z ---------------------------------------------------------------------- 2022-09-27T16:20:32.3391552Z test_init_sharded_tensor_with_kaiming_uniform (__main__.TestShardedTensorNNInit) 2022-09-27T16:20:32.3392482Z Test torch.nn.init.kaiming_uniform_(ShardedTensor, a, mode, nonlinearit) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:32.3402144Z test_init_sharded_tensor_with_normal (__main__.TestShardedTensorNNInit) 2022-09-27T16:20:32.3402623Z Test torch.nn.init.normal_(ShardedTensor, mean, std) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:32.3412196Z test_init_sharded_tensor_with_uniform (__main__.TestShardedTensorNNInit) 2022-09-27T16:20:32.3412882Z Test torch.nn.init.uniform_(ShardedTensor, a, b) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:32.3413287Z 2022-09-27T16:20:32.3413700Z ---------------------------------------------------------------------- 2022-09-27T16:20:32.3413936Z Ran 3 tests in 0.003s 2022-09-27T16:20:32.3414054Z 2022-09-27T16:20:32.3414124Z OK (skipped=3) 2022-09-27T16:20:32.3414232Z 2022-09-27T16:20:32.3414318Z Generating XML reports... 2022-09-27T16:20:32.3439581Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20220927162032.xml 2022-09-27T16:20:32.5612765Z Running distributed/test_c10d_spawn_nccl ... [2022-09-27 16:20:32.560903] 2022-09-27T16:20:32.5613597Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:32.560990] 2022-09-27T16:20:33.7662356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxr2w10g 2022-09-27T16:20:33.7663309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxr2w10g/_remote_module_non_scriptable.py 2022-09-27T16:20:33.9986396Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_reduce_nccl>]> 2022-09-27T16:20:33.9987239Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) 2022-09-27T16:20:33.9987563Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) 2022-09-27T16:20:33.9987889Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) 2022-09-27T16:20:33.9988200Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) 2022-09-27T16:20:33.9988462Z 2022-09-27T16:20:33.9988696Z 2022-09-27T16:20:33.9989708Z , <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather_base>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter_non_contiguous>]> 2022-09-27T16:20:33.9990764Z test_all_gather (__main__.TestDistributedNNFunctionsNccl) 2022-09-27T16:20:33.9991069Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) 2022-09-27T16:20:33.9991370Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl) 2022-09-27T16:20:33.9991658Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) 2022-09-27T16:20:33.9991960Z test_allreduce (__main__.TestDistributedNNFunctionsNccl) 2022-09-27T16:20:33.9992260Z test_broadcast (__main__.TestDistributedNNFunctionsNccl) 2022-09-27T16:20:33.9992539Z test_reduce (__main__.TestDistributedNNFunctionsNccl) 2022-09-27T16:20:33.9992836Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) 2022-09-27T16:20:33.9993158Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) 2022-09-27T16:20:35.0274800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:35.0275179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:35.0282927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:35.0283313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:35.1849259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8698xdp 2022-09-27T16:20:35.1850194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8698xdp/_remote_module_non_scriptable.py 2022-09-27T16:20:35.4306634Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-09-27T16:20:35.4322354Z 2022-09-27T16:20:35.4322632Z Running tests... 2022-09-27T16:20:35.4322984Z ---------------------------------------------------------------------- 2022-09-27T16:20:35.4328457Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-09-27T16:20:35.4328847Z 2022-09-27T16:20:35.4329684Z ---------------------------------------------------------------------- 2022-09-27T16:20:35.4330147Z Ran 1 test in 0.001s 2022-09-27T16:20:35.4330342Z 2022-09-27T16:20:35.4330462Z OK (skipped=1) 2022-09-27T16:20:35.4330765Z 2022-09-27T16:20:35.4330898Z Generating XML reports... 2022-09-27T16:20:35.4354657Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220927162035.xml 2022-09-27T16:20:36.7273368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:36.7273730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:36.7281596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:36.7281954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:36.8814969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp93_yqlp 2022-09-27T16:20:36.8816041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp93_yqlp/_remote_module_non_scriptable.py 2022-09-27T16:20:37.1086118Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-09-27T16:20:37.1102045Z 2022-09-27T16:20:37.1102116Z Running tests... 2022-09-27T16:20:37.1102661Z ---------------------------------------------------------------------- 2022-09-27T16:20:37.1107975Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-09-27T16:20:37.1108346Z 2022-09-27T16:20:37.1108657Z ---------------------------------------------------------------------- 2022-09-27T16:20:37.1108890Z Ran 1 test in 0.001s 2022-09-27T16:20:37.1109005Z 2022-09-27T16:20:37.1109080Z OK (skipped=1) 2022-09-27T16:20:37.1109191Z 2022-09-27T16:20:37.1109275Z Generating XML reports... 2022-09-27T16:20:37.1133082Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220927162037.xml 2022-09-27T16:20:38.4198507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:38.4198927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:38.4207120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:38.4207547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:38.5782021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8lauf1r_ 2022-09-27T16:20:38.5783012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8lauf1r_/_remote_module_non_scriptable.py 2022-09-27T16:20:38.8117378Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-09-27T16:20:38.8133562Z 2022-09-27T16:20:38.8133981Z Running tests... 2022-09-27T16:20:38.8134428Z ---------------------------------------------------------------------- 2022-09-27T16:20:38.8139498Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-09-27T16:20:38.8139828Z 2022-09-27T16:20:38.8140138Z ---------------------------------------------------------------------- 2022-09-27T16:20:38.8140373Z Ran 1 test in 0.001s 2022-09-27T16:20:38.8140489Z 2022-09-27T16:20:38.8140566Z OK (skipped=1) 2022-09-27T16:20:38.8140676Z 2022-09-27T16:20:38.8140763Z Generating XML reports... 2022-09-27T16:20:38.8164512Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220927162038.xml 2022-09-27T16:20:40.1240566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:40.1240926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:40.1249373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:40.1249738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:40.2831038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31q8hd5d 2022-09-27T16:20:40.2831814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31q8hd5d/_remote_module_non_scriptable.py 2022-09-27T16:20:40.5207140Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-09-27T16:20:40.5222671Z 2022-09-27T16:20:40.5222815Z Running tests... 2022-09-27T16:20:40.5223290Z ---------------------------------------------------------------------- 2022-09-27T16:20:40.5228677Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-09-27T16:20:40.5229044Z 2022-09-27T16:20:40.5229441Z ---------------------------------------------------------------------- 2022-09-27T16:20:40.5229868Z Ran 1 test in 0.001s 2022-09-27T16:20:40.5230006Z 2022-09-27T16:20:40.5230080Z OK (skipped=1) 2022-09-27T16:20:40.5230189Z 2022-09-27T16:20:40.5230267Z Generating XML reports... 2022-09-27T16:20:40.5255375Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220927162040.xml 2022-09-27T16:20:41.8426136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:41.8426523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:41.8435019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:41.8435367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:42.0018371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkc6ammzo 2022-09-27T16:20:42.0019407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkc6ammzo/_remote_module_non_scriptable.py 2022-09-27T16:20:42.2333893Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-09-27T16:20:42.2349712Z 2022-09-27T16:20:42.2349807Z Running tests... 2022-09-27T16:20:42.2350761Z ---------------------------------------------------------------------- 2022-09-27T16:20:42.2354365Z test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:20:42.2355070Z 2022-09-27T16:20:42.2355559Z ---------------------------------------------------------------------- 2022-09-27T16:20:42.2355974Z Ran 1 test in 0.000s 2022-09-27T16:20:42.2356171Z 2022-09-27T16:20:42.2356288Z OK (skipped=1) 2022-09-27T16:20:42.2356471Z 2022-09-27T16:20:42.2356609Z Generating XML reports... 2022-09-27T16:20:42.2380324Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162042.xml 2022-09-27T16:20:43.5365043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:43.5365422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:43.5373186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:43.5373550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:43.6961318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6irnmpv0 2022-09-27T16:20:43.6962467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6irnmpv0/_remote_module_non_scriptable.py 2022-09-27T16:20:43.9337519Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-09-27T16:20:43.9353981Z 2022-09-27T16:20:43.9354157Z Running tests... 2022-09-27T16:20:43.9354776Z ---------------------------------------------------------------------- 2022-09-27T16:20:43.9369077Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:43.9369678Z 2022-09-27T16:20:43.9370012Z ---------------------------------------------------------------------- 2022-09-27T16:20:43.9370264Z Ran 1 test in 0.001s 2022-09-27T16:20:43.9370379Z 2022-09-27T16:20:43.9370439Z OK (skipped=1) 2022-09-27T16:20:43.9370550Z 2022-09-27T16:20:43.9370636Z Generating XML reports... 2022-09-27T16:20:43.9393822Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162043.xml 2022-09-27T16:20:45.2627195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:45.2627577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:45.2635742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:45.2636106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:45.4202463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0q8v7sb 2022-09-27T16:20:45.4203630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0q8v7sb/_remote_module_non_scriptable.py 2022-09-27T16:20:45.6537136Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-09-27T16:20:45.6552973Z 2022-09-27T16:20:45.6553117Z Running tests... 2022-09-27T16:20:45.6553696Z ---------------------------------------------------------------------- 2022-09-27T16:20:45.6557866Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:20:45.6558347Z 2022-09-27T16:20:45.6558812Z ---------------------------------------------------------------------- 2022-09-27T16:20:45.6559240Z Ran 1 test in 0.000s 2022-09-27T16:20:45.6559376Z 2022-09-27T16:20:45.6559472Z OK (skipped=1) 2022-09-27T16:20:45.6559581Z 2022-09-27T16:20:45.6559664Z Generating XML reports... 2022-09-27T16:20:45.6582192Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162045.xml 2022-09-27T16:20:46.9630727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:46.9631147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:46.9639670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:46.9640023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:47.1220632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphi3iruyh 2022-09-27T16:20:47.1222104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphi3iruyh/_remote_module_non_scriptable.py 2022-09-27T16:20:47.3540762Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-09-27T16:20:47.3557616Z 2022-09-27T16:20:47.3557915Z Running tests... 2022-09-27T16:20:47.3558537Z ---------------------------------------------------------------------- 2022-09-27T16:20:47.3563160Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:20:47.3563470Z 2022-09-27T16:20:47.3563731Z ---------------------------------------------------------------------- 2022-09-27T16:20:47.3563976Z Ran 1 test in 0.001s 2022-09-27T16:20:47.3564090Z 2022-09-27T16:20:47.3564161Z OK (skipped=1) 2022-09-27T16:20:47.3564271Z 2022-09-27T16:20:47.3564344Z Generating XML reports... 2022-09-27T16:20:47.3588879Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162047.xml 2022-09-27T16:20:48.6657461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:48.6657849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:48.6666706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:48.6667087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:48.8271941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp295ohmgh 2022-09-27T16:20:48.8278288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp295ohmgh/_remote_module_non_scriptable.py 2022-09-27T16:20:49.0572682Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-09-27T16:20:49.0588467Z 2022-09-27T16:20:49.0588601Z Running tests... 2022-09-27T16:20:49.0589192Z ---------------------------------------------------------------------- 2022-09-27T16:20:49.0593565Z test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:20:49.0593886Z 2022-09-27T16:20:49.0594292Z ---------------------------------------------------------------------- 2022-09-27T16:20:49.0594898Z Ran 1 test in 0.001s 2022-09-27T16:20:49.0595043Z 2022-09-27T16:20:49.0595116Z OK (skipped=1) 2022-09-27T16:20:49.0595223Z 2022-09-27T16:20:49.0595312Z Generating XML reports... 2022-09-27T16:20:49.0618454Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162049.xml 2022-09-27T16:20:50.3696170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:50.3696549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:50.3704038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:50.3704416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:50.5265333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsfkl9oa9 2022-09-27T16:20:50.5266461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsfkl9oa9/_remote_module_non_scriptable.py 2022-09-27T16:20:50.7627865Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-09-27T16:20:50.7645037Z 2022-09-27T16:20:50.7645156Z Running tests... 2022-09-27T16:20:50.7645847Z ---------------------------------------------------------------------- 2022-09-27T16:20:50.7649810Z test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:20:50.7650148Z 2022-09-27T16:20:50.7650509Z ---------------------------------------------------------------------- 2022-09-27T16:20:50.7650947Z Ran 1 test in 0.000s 2022-09-27T16:20:50.7651112Z 2022-09-27T16:20:50.7651193Z OK (skipped=1) 2022-09-27T16:20:50.7651306Z 2022-09-27T16:20:50.7651389Z Generating XML reports... 2022-09-27T16:20:50.7677076Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162050.xml 2022-09-27T16:20:52.0676192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:52.0676618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:52.0684484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:52.0685032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:52.2246795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppu7i85j4 2022-09-27T16:20:52.2247582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppu7i85j4/_remote_module_non_scriptable.py 2022-09-27T16:20:52.4602286Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-09-27T16:20:52.4618236Z 2022-09-27T16:20:52.4618508Z Running tests... 2022-09-27T16:20:52.4619224Z ---------------------------------------------------------------------- 2022-09-27T16:20:52.4622737Z test_reduce (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:20:52.4623170Z 2022-09-27T16:20:52.4623756Z ---------------------------------------------------------------------- 2022-09-27T16:20:52.4624187Z Ran 1 test in 0.000s 2022-09-27T16:20:52.4624381Z 2022-09-27T16:20:52.4624504Z OK (skipped=1) 2022-09-27T16:20:52.4624672Z 2022-09-27T16:20:52.4624816Z Generating XML reports... 2022-09-27T16:20:52.4649388Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162052.xml 2022-09-27T16:20:53.7645447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:53.7646044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:53.7654346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:53.7665129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:53.9216724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptmiugsb_ 2022-09-27T16:20:53.9217410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptmiugsb_/_remote_module_non_scriptable.py 2022-09-27T16:20:54.1513155Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-09-27T16:20:54.1529715Z 2022-09-27T16:20:54.1530136Z Running tests... 2022-09-27T16:20:54.1530548Z ---------------------------------------------------------------------- 2022-09-27T16:20:54.1544312Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:54.1544690Z 2022-09-27T16:20:54.1545091Z ---------------------------------------------------------------------- 2022-09-27T16:20:54.1545495Z Ran 1 test in 0.001s 2022-09-27T16:20:54.1545613Z 2022-09-27T16:20:54.1545683Z OK (skipped=1) 2022-09-27T16:20:54.1545792Z 2022-09-27T16:20:54.1545875Z Generating XML reports... 2022-09-27T16:20:54.1569760Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162054.xml 2022-09-27T16:20:55.4650863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests 2022-09-27T16:20:55.4651234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-09-27T16:20:55.4658995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests 2022-09-27T16:20:55.4659475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-09-27T16:20:55.6237952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjh5bor_l 2022-09-27T16:20:55.6239131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjh5bor_l/_remote_module_non_scriptable.py 2022-09-27T16:20:55.8567402Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-09-27T16:20:55.8582687Z 2022-09-27T16:20:55.8582811Z Running tests... 2022-09-27T16:20:55.8583384Z ---------------------------------------------------------------------- 2022-09-27T16:20:55.8595868Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:55.8596300Z 2022-09-27T16:20:55.8596691Z ---------------------------------------------------------------------- 2022-09-27T16:20:55.8597046Z Ran 1 test in 0.001s 2022-09-27T16:20:55.8597158Z 2022-09-27T16:20:55.8597227Z OK (skipped=1) 2022-09-27T16:20:55.8597557Z 2022-09-27T16:20:55.8597648Z Generating XML reports... 2022-09-27T16:20:55.8619976Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162055.xml 2022-09-27T16:20:56.4590636Z Running distributed/_shard/sharded_tensor/ops/test_embedding ... [2022-09-27 16:20:56.458690] 2022-09-27T16:20:56.4591274Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:56.458769] 2022-09-27T16:20:57.6802677Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding 2022-09-27T16:20:57.6814379Z 2022-09-27T16:20:57.6814601Z Running tests... 2022-09-27T16:20:57.6815000Z ---------------------------------------------------------------------- 2022-09-27T16:20:57.6827440Z test_sharded_embedding_colwise (__main__.TestShardedEmbedding) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:57.6840215Z test_sharded_embedding_rowwise (__main__.TestShardedEmbedding) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:57.6840683Z 2022-09-27T16:20:57.6841084Z ---------------------------------------------------------------------- 2022-09-27T16:20:57.6841484Z Ran 2 tests in 0.003s 2022-09-27T16:20:57.6841685Z 2022-09-27T16:20:57.6841800Z OK (skipped=2) 2022-09-27T16:20:57.6841967Z 2022-09-27T16:20:57.6842083Z Generating XML reports... 2022-09-27T16:20:57.6867635Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20220927162057.xml 2022-09-27T16:20:57.8932871Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard ... [2022-09-27 16:20:57.892847] 2022-09-27T16:20:59.0894998Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:57.892930] 2022-09-27T16:20:59.0896072Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard 2022-09-27T16:20:59.0906646Z 2022-09-27T16:20:59.0907134Z Running tests... 2022-09-27T16:20:59.0907580Z ---------------------------------------------------------------------- 2022-09-27T16:20:59.0914983Z test_sharded_tensor_reshard (__main__.TestReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:59.0924444Z test_sharded_tensor_reshard_errors (__main__.TestReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-09-27T16:20:59.0925285Z 2022-09-27T16:20:59.0925595Z ---------------------------------------------------------------------- 2022-09-27T16:20:59.0926039Z Ran 2 tests in 0.002s 2022-09-27T16:20:59.0926239Z 2022-09-27T16:20:59.0926367Z OK (skipped=2) 2022-09-27T16:20:59.0926559Z 2022-09-27T16:20:59.0926712Z Generating XML reports... 2022-09-27T16:20:59.0951290Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20220927162059.xml 2022-09-27T16:20:59.3130724Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2022-09-27 16:20:59.312669] 2022-09-27T16:20:59.3131400Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:59.312752] 2022-09-27T16:21:00.5147593Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag 2022-09-27T16:21:00.5158575Z 2022-09-27T16:21:00.5158674Z Running tests... 2022-09-27T16:21:00.5159438Z ---------------------------------------------------------------------- 2022-09-27T16:21:00.5163754Z test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:21:00.5167148Z test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:21:00.5167510Z 2022-09-27T16:21:00.5167791Z ---------------------------------------------------------------------- 2022-09-27T16:21:00.5168061Z Ran 2 tests in 0.001s 2022-09-27T16:21:00.5168176Z 2022-09-27T16:21:00.5168237Z OK (skipped=2) 2022-09-27T16:21:00.5168342Z 2022-09-27T16:21:00.5168426Z Generating XML reports... 2022-09-27T16:21:00.5193670Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220927162100.xml 2022-09-27T16:21:00.7398289Z Running distributed/_shard/sharded_tensor/ops/test_softmax ... [2022-09-27 16:21:00.739413] 2022-09-27T16:21:00.7398997Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_softmax.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:21:00.739510] 2022-09-27T16:21:01.9371783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahkc1b3j 2022-09-27T16:21:01.9372717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahkc1b3j/_remote_module_non_scriptable.py 2022-09-27T16:21:01.9489998Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax 2022-09-27T16:21:01.9502806Z 2022-09-27T16:21:01.9503113Z Running tests... 2022-09-27T16:21:01.9503724Z ---------------------------------------------------------------------- 2022-09-27T16:21:01.9508481Z test_sharded_softmax_basic (__main__.TestShardedSoftmax) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:21:01.9513231Z test_sharded_softmax_on_sharding_dim (__main__.TestShardedSoftmax) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-09-27T16:21:01.9513648Z 2022-09-27T16:21:01.9514043Z ---------------------------------------------------------------------- 2022-09-27T16:21:01.9514452Z Ran 2 tests in 0.001s 2022-09-27T16:21:01.9514893Z 2022-09-27T16:21:01.9515015Z OK (skipped=2) 2022-09-27T16:21:01.9515182Z 2022-09-27T16:21:01.9515347Z Generating XML reports... 2022-09-27T16:21:01.9548509Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20220927162101.xml 2022-09-27T16:21:02.1684329Z Running distributed/_shard/sharded_tensor/ops/test_math_ops ... [2022-09-27 16:21:02.168071] 2022-09-27T16:21:02.1684947Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:21:02.168151] 2022-09-27T16:21:03.5773513Z Running distributed/_shard/test_sharder ... [2022-09-27 16:21:03.576885] 2022-09-27T16:21:03.5774109Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:21:03.576969] 2022-09-27T16:21:04.7586419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnrggkp8 2022-09-27T16:21:04.7587318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnrggkp8/_remote_module_non_scriptable.py 2022-09-27T16:21:04.9671073Z Running distributed/elastic/timer/api_test ... [2022-09-27 16:21:04.966673] 2022-09-27T16:21:04.9671695Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:21:04.966760] 2022-09-27T16:21:06.1751788Z Running distributed/pipeline/sync/skip/test_api ... [2022-09-27 16:21:06.174781] 2022-09-27T16:21:06.1752300Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2022-09-27 16:21:06.174866] 2022-09-27T16:21:07.8935536Z ============================= test session starts ============================== 2022-09-27T16:21:07.8936199Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-09-27T16:21:07.9513512Z cachedir: .pytest_cache 2022-09-27T16:21:07.9514418Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-09-27T16:21:07.9515058Z torch: 1.13.0a0+git52424e2 2022-09-27T16:21:07.9515307Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-09-27T16:21:07.9515723Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2 2022-09-27T16:21:07.9621529Z collecting ...  2022-09-27T16:21:07.9622075Z collected 3 items  2022-09-27T16:21:07.9624862Z 2022-09-27T16:21:07.9649708Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%] 2022-09-27T16:21:07.9661273Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED [ 66%] 2022-09-27T16:21:07.9684052Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED [100%] 2022-09-27T16:21:07.9684808Z 2022-09-27T16:21:07.9685203Z ============================== 3 passed in 0.08s =============================== 2022-09-27T16:21:08.1929796Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-09-27 16:21:08.192609] 2022-09-27T16:21:08.1930545Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-09-27 16:21:08.192693] 2022-09-27T16:21:09.5680041Z ============================= test session starts ============================== 2022-09-27T16:21:09.5680509Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-09-27T16:21:09.5751969Z cachedir: .pytest_cache 2022-09-27T16:21:09.5752565Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-09-27T16:21:09.5752932Z torch: 1.13.0a0+git52424e2 2022-09-27T16:21:09.5753235Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-09-27T16:21:09.5753646Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2 2022-09-27T16:21:09.5882340Z collecting ...  2022-09-27T16:21:09.5882657Z collected 6 items  2022-09-27T16:21:09.5886026Z 2022-09-27T16:21:09.5911544Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%] 2022-09-27T16:21:09.5924438Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%] 2022-09-27T16:21:09.5936632Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%] 2022-09-27T16:21:09.5948689Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%] 2022-09-27T16:21:09.5963063Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%] 2022-09-27T16:21:09.5992498Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%] 2022-09-27T16:21:09.5993759Z 2022-09-27T16:21:09.5994041Z ============================== 6 passed in 0.03s =============================== 2022-09-27T16:21:09.8523758Z Running distributed/pipeline/sync/skip/test_portal ... [2022-09-27 16:21:09.851971] 2022-09-27T16:21:09.8524285Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-09-27 16:21:09.852053] 2022-09-27T16:21:11.2460383Z ============================= test session starts ============================== 2022-09-27T16:21:11.2460824Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-09-27T16:21:11.2537392Z cachedir: .pytest_cache 2022-09-27T16:21:11.2538139Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-09-27T16:21:11.2538571Z torch: 1.13.0a0+git52424e2 2022-09-27T16:21:11.2538816Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-09-27T16:21:11.2539207Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2 2022-09-27T16:21:11.2772311Z collecting ...  2022-09-27T16:21:11.2772881Z collected 10 items  2022-09-27T16:21:11.2775790Z 2022-09-27T16:21:11.2786292Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device SKIPPED [ 10%] 2022-09-27T16:21:11.2821725Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED [ 20%] 2022-09-27T16:21:11.2837158Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%] 2022-09-27T16:21:11.2849457Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED [ 40%] 2022-09-27T16:21:11.2862424Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%] 2022-09-27T16:21:11.2875823Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%] 2022-09-27T16:21:11.2888675Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%] 2022-09-27T16:21:11.2901805Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%] 2022-09-27T16:21:11.2915308Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%] 2022-09-27T16:21:11.2931764Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%] 2022-09-27T16:21:11.2932222Z 2022-09-27T16:21:11.2932481Z ========================= 9 passed, 1 skipped in 0.05s ========================= 2022-09-27T16:21:11.5558115Z Running distributed/pipeline/sync/skip/test_tracker ... [2022-09-27 16:21:11.555431] 2022-09-27T16:21:11.5558697Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2022-09-27 16:21:11.555529] 2022-09-27T16:21:12.9358802Z ============================= test session starts ============================== 2022-09-27T16:21:12.9359245Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-09-27T16:21:12.9429719Z cachedir: .pytest_cache 2022-09-27T16:21:12.9430660Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-09-27T16:21:12.9431028Z torch: 1.13.0a0+git52424e2 2022-09-27T16:21:12.9431291Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-09-27T16:21:12.9431704Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2 2022-09-27T16:21:12.9591919Z collecting ...  2022-09-27T16:21:12.9592460Z collected 6 items  2022-09-27T16:21:12.9595116Z 2022-09-27T16:21:12.9621534Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%] 2022-09-27T16:21:12.9629964Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel SKIPPED [ 33%] 2022-09-27T16:21:12.9645057Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%] 2022-09-27T16:21:12.9657039Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%] 2022-09-27T16:21:12.9670558Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%] 2022-09-27T16:21:12.9684998Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%] 2022-09-27T16:21:12.9685867Z 2022-09-27T16:21:12.9686283Z ========================= 5 passed, 1 skipped in 0.03s ========================= 2022-09-27T16:21:13.1958417Z Running distributed/pipeline/sync/test_balance ... [2022-09-27 16:21:13.195423] 2022-09-27T16:21:13.1958921Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-09-27 16:21:13.195516] 2022-09-27T16:21:14.5847124Z ============================= test session starts ============================== 2022-09-27T16:21:14.5847566Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-09-27T16:21:14.5922302Z cachedir: .pytest_cache 2022-09-27T16:21:14.5923034Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-09-27T16:21:14.5923372Z torch: 1.13.0a0+git52424e2 2022-09-27T16:21:14.5923619Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-09-27T16:21:14.5924019Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2 2022-09-27T16:21:14.6133991Z collecting ...  2022-09-27T16:21:14.6134293Z collected 15 items  2022-09-27T16:21:14.6137121Z 2022-09-27T16:21:14.6160749Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED [ 6%] 2022-09-27T16:21:14.6171789Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 13%] 2022-09-27T16:21:14.6183251Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 20%] 2022-09-27T16:21:14.6194779Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 26%] 2022-09-27T16:21:14.6203001Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 33%] 2022-09-27T16:21:15.6244179Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 40%] 2022-09-27T16:21:15.6252441Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent SKIPPED [ 46%] 2022-09-27T16:21:15.6260645Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param SKIPPED [ 53%] 2022-09-27T16:21:15.6269048Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale SKIPPED [ 60%] 2022-09-27T16:21:15.6292580Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 66%] 2022-09-27T16:21:16.6322045Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 73%] 2022-09-27T16:21:17.6363188Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED [ 80%] 2022-09-27T16:21:18.6396319Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 86%] 2022-09-27T16:21:18.6404387Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple SKIPPED [ 93%] 2022-09-27T16:21:18.6431126Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED [100%] 2022-09-27T16:21:18.6432512Z 2022-09-27T16:21:18.6432822Z ======================== 10 passed, 5 skipped in 4.06s ========================= 2022-09-27T16:21:18.8836937Z Running distributed/pipeline/sync/test_checkpoint ... [2022-09-27 16:21:18.883353] 2022-09-27T16:21:18.8837436Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-09-27 16:21:18.883434] 2022-09-27T16:21:20.2716100Z ============================= test session starts ============================== 2022-09-27T16:21:20.2716551Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-09-27T16:21:20.2788483Z cachedir: .pytest_cache 2022-09-27T16:21:20.2789053Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-09-27T16:21:20.2789386Z torch: 1.13.0a0+git52424e2 2022-09-27T16:21:20.2790526Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-09-27T16:21:20.2791353Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2 2022-09-27T16:21:20.2932194Z collecting ...  2022-09-27T16:21:20.2932738Z collected 7 items  2022-09-27T16:21:20.2935514Z 2022-09-27T16:21:20.2978532Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 14%] 2022-09-27T16:21:20.2993917Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 28%] 2022-09-27T16:21:20.3009045Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 42%] 2022-09-27T16:21:20.3037778Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 57%] 2022-09-27T16:21:20.3052537Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 71%] 2022-09-27T16:21:20.3065232Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 85%] 2022-09-27T16:21:20.3083518Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%] 2022-09-27T16:21:20.3084463Z 2022-09-27T16:21:20.3084986Z ============================== 7 passed in 0.04s =============================== 2022-09-27T16:21:20.5270832Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2022-09-27 16:21:20.526653] 2022-09-27T16:21:20.5271388Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2022-09-27 16:21:20.526732] 2022-09-27T16:21:21.8944223Z ============================= test session starts ============================== 2022-09-27T16:21:21.8944713Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-09-27T16:21:21.9015670Z cachedir: .pytest_cache 2022-09-27T16:21:21.9016335Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-09-27T16:21:21.9016807Z torch: 1.13.0a0+git52424e2 2022-09-27T16:21:21.9017176Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-09-27T16:21:21.9017896Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2 2022-09-27T16:21:21.9228818Z collecting ...  2022-09-27T16:21:21.9229175Z collected 11 items  2022-09-27T16:21:21.9232299Z 2022-09-27T16:21:22.0030816Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [ 9%] 2022-09-27T16:21:22.0535907Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%] 2022-09-27T16:21:22.0882113Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%] 2022-09-27T16:21:22.1265067Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%] 2022-09-27T16:21:22.1488037Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%] 2022-09-27T16:21:22.1705189Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%] 2022-09-27T16:21:22.1722672Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%] 2022-09-27T16:21:22.2085462Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED [ 72%] 2022-09-27T16:21:22.3608807Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%] 2022-09-27T16:21:22.5249671Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%] 2022-09-27T16:21:22.5445246Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%] 2022-09-27T16:21:22.5446027Z 2022-09-27T16:21:22.5447126Z ============================== 11 passed in 0.65s ============================== 2022-09-27T16:21:22.7928291Z Running distributed/pipeline/sync/test_inplace ... [2022-09-27 16:21:22.792472] 2022-09-27T16:21:22.7928884Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-09-27 16:21:22.792551] 2022-09-27T16:21:24.1825989Z ============================= test session starts ============================== 2022-09-27T16:21:24.1826565Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-09-27T16:21:24.1909840Z cachedir: .pytest_cache 2022-09-27T16:21:24.1910886Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-09-27T16:21:24.1911499Z torch: 1.13.0a0+git52424e2 2022-09-27T16:21:24.1911931Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-09-27T16:21:24.1912346Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2 2022-09-27T16:21:24.2001087Z collecting ...  2022-09-27T16:21:24.2001667Z collected 3 items  2022-09-27T16:21:24.2004671Z 2022-09-27T16:21:24.2455646Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%] 2022-09-27T16:21:24.2652304Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%] 2022-09-27T16:21:24.2855209Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%] 2022-09-27T16:21:24.2855678Z 2022-09-27T16:21:24.2855980Z ========================= 1 passed, 2 xfailed in 0.10s ========================= 2022-09-27T16:21:24.5208043Z Running distributed/pipeline/sync/test_phony ... [2022-09-27 16:21:24.520428] 2022-09-27T16:21:24.5208549Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-09-27 16:21:24.520510] 2022-09-27T16:21:25.8945220Z ============================= test session starts ============================== 2022-09-27T16:21:25.8945972Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-09-27T16:21:25.9017634Z cachedir: .pytest_cache 2022-09-27T16:21:25.9018521Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-09-27T16:21:25.9019107Z torch: 1.13.0a0+git52424e2 2022-09-27T16:21:25.9019498Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-09-27T16:21:25.9020189Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2 2022-09-27T16:21:25.9150909Z collecting ...  2022-09-27T16:21:25.9151243Z collected 4 items  2022-09-27T16:21:25.9154732Z 2022-09-27T16:21:25.9179518Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED [ 25%] 2022-09-27T16:21:25.9191479Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%] 2022-09-27T16:21:25.9202788Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED [ 75%] 2022-09-27T16:21:25.9220363Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%] 2022-09-27T16:21:25.9221577Z 2022-09-27T16:21:25.9221875Z ============================== 4 passed in 0.03s =============================== 2022-09-27T16:21:26.1442312Z Running distributed/pipeline/sync/test_pipeline ... [2022-09-27 16:21:26.143852] 2022-09-27T16:21:26.1442832Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2022-09-27 16:21:26.143936] 2022-09-27T16:21:27.5207208Z ============================= test session starts ============================== 2022-09-27T16:21:27.5207644Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-09-27T16:21:27.5277976Z cachedir: .pytest_cache 2022-09-27T16:21:27.5278803Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-09-27T16:21:27.5279196Z torch: 1.13.0a0+git52424e2 2022-09-27T16:21:27.5279440Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-09-27T16:21:27.5279845Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2 2022-09-27T16:21:27.5400866Z collecting ...  2022-09-27T16:21:27.5401240Z collected 1 item  2022-09-27T16:21:27.5404401Z 2022-09-27T16:21:27.5430747Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED [100%] 2022-09-27T16:21:27.5431962Z 2022-09-27T16:21:27.5432324Z ============================== 1 passed in 0.02s =============================== 2022-09-27T16:21:27.7837336Z Running distributed/pipeline/sync/test_transparency ... [2022-09-27 16:21:27.783365] 2022-09-27T16:21:27.7837911Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-09-27 16:21:27.783468] 2022-09-27T16:21:29.1599971Z ============================= test session starts ============================== 2022-09-27T16:21:29.1600406Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python 2022-09-27T16:21:29.1670433Z cachedir: .pytest_cache 2022-09-27T16:21:29.1671299Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-09-27T16:21:29.1671724Z torch: 1.13.0a0+git52424e2 2022-09-27T16:21:29.1671956Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-09-27T16:21:29.1672358Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2 2022-09-27T16:21:29.1754320Z collecting ...  2022-09-27T16:21:29.1754806Z collected 1 item  2022-09-27T16:21:29.1757947Z 2022-09-27T16:21:29.2289337Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%] 2022-09-27T16:21:29.2289603Z 2022-09-27T16:21:29.2290271Z ============================== 1 passed in 0.07s =============================== 2022-09-27T16:21:29.4503023Z Running distributed/test_nccl ... [2022-09-27 16:21:29.449913] 2022-09-27T16:21:29.4503802Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:21:29.449994] 2022-09-27T16:21:30.8609755Z CUDA not available, skipping tests 2022-09-27T16:21:30.8630845Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl 2022-09-27T16:21:30.8642496Z 2022-09-27T16:21:30.8642769Z Running tests... 2022-09-27T16:21:30.8643305Z ---------------------------------------------------------------------- 2022-09-27T16:21:30.8643523Z 2022-09-27T16:21:30.8643766Z ---------------------------------------------------------------------- 2022-09-27T16:21:30.8644222Z Ran 0 tests in 0.000s 2022-09-27T16:21:30.8644339Z 2022-09-27T16:21:30.8644399Z OK 2022-09-27T16:21:30.8644556Z 2022-09-27T16:21:30.8644645Z Generating XML reports... 2022-09-27T16:21:31.4547894Z 2022-09-27T16:21:31.4548366Z real 52m27.469s 2022-09-27T16:21:31.4548817Z user 110m52.264s 2022-09-27T16:21:31.4549091Z sys 26m33.513s 2022-09-27T16:21:31.4549278Z + assert_git_not_dirty 2022-09-27T16:21:31.4549686Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]] 2022-09-27T16:21:31.4550048Z + [[ linux-focal-py3.7-gcc7 != *xla* ]] 2022-09-27T16:21:31.4551003Z ++ git status --porcelain 2022-09-27T16:21:32.7124154Z + git_status= 2022-09-27T16:21:32.7124561Z + [[ -n '' ]] 2022-09-27T16:21:32.7124871Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]] 2022-09-27T16:21:32.7125071Z + [[ 1 == 1 ]] 2022-09-27T16:21:32.7125234Z + test_rpc 2022-09-27T16:21:32.7125514Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]] 2022-09-27T16:21:32.7125770Z + echo 'Testing RPC C++ tests' 2022-09-27T16:21:32.7125986Z Testing RPC C++ tests 2022-09-27T16:21:32.7126862Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.7/site-packages/torch/bin 2022-09-27T16:21:32.7135438Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.7/site-packages/torch/bin 2022-09-27T16:21:32.7143771Z + ln -sf '/opt/conda/lib/python3.7/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.7/site-packages/torch/bin 2022-09-27T16:21:32.7151393Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc 2022-09-27T16:21:32.7151718Z + mkdir -p test/test-reports/cpp-rpc/test_rpc 2022-09-27T16:21:32.7163244Z + /opt/conda/lib/python3.7/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml 2022-09-27T16:21:32.9745546Z CUDA not available. Disabling CUDA and MultiCUDA tests 2022-09-27T16:21:32.9746047Z Note: Google Test filter = *-*_CUDA:*_MultiCUDA 2022-09-27T16:21:32.9746355Z [==========] Running 8 tests from 3 test suites. 2022-09-27T16:21:32.9746675Z [----------] Global test environment set-up. 2022-09-27T16:21:32.9746956Z [----------] 4 tests from WireSerialize 2022-09-27T16:21:32.9747227Z [ RUN ] WireSerialize.Base 2022-09-27T16:21:32.9923037Z [ OK ] WireSerialize.Base (17 ms) 2022-09-27T16:21:32.9923407Z [ RUN ] WireSerialize.RecopySparseTensors 2022-09-27T16:21:33.0002767Z [ OK ] WireSerialize.RecopySparseTensors (7 ms) 2022-09-27T16:21:33.0003329Z [ RUN ] WireSerialize.CloneSparseTensors 2022-09-27T16:21:33.0078413Z [ OK ] WireSerialize.CloneSparseTensors (7 ms) 2022-09-27T16:21:33.0078955Z [ RUN ] WireSerialize.Errors 2022-09-27T16:21:33.0102190Z [ OK ] WireSerialize.Errors (2 ms) 2022-09-27T16:21:33.0102803Z [----------] 4 tests from WireSerialize (35 ms total) 2022-09-27T16:21:33.0103085Z 2022-09-27T16:21:33.0103259Z [----------] 1 test from TestE2ETensorPipe 2022-09-27T16:21:33.0103576Z [ RUN ] TestE2ETensorPipe.TestTrainingLoop 2022-09-27T16:21:33.3906444Z [W tensorpipe_agent.cpp:726] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-09-27T16:21:33.3916887Z [ OK ] TestE2ETensorPipe.TestTrainingLoop (381 ms) 2022-09-27T16:21:33.3917332Z [----------] 1 test from TestE2ETensorPipe (381 ms total) 2022-09-27T16:21:33.3917570Z 2022-09-27T16:21:33.3917749Z [----------] 3 tests from TensorpipeSerialize 2022-09-27T16:21:33.3918038Z [ RUN ] TensorpipeSerialize.Base 2022-09-27T16:21:33.3918561Z [ OK ] TensorpipeSerialize.Base (0 ms) 2022-09-27T16:21:33.3918907Z [ RUN ] TensorpipeSerialize.RecopySparseTensors 2022-09-27T16:21:33.3987296Z [ OK ] TensorpipeSerialize.RecopySparseTensors (6 ms) 2022-09-27T16:21:33.3988024Z [ RUN ] TensorpipeSerialize.NoDeleterTensors 2022-09-27T16:21:33.3988722Z [ OK ] TensorpipeSerialize.NoDeleterTensors (0 ms) 2022-09-27T16:21:33.3989392Z [----------] 3 tests from TensorpipeSerialize (7 ms total) 2022-09-27T16:21:33.3989697Z 2022-09-27T16:21:33.3990000Z [----------] Global test environment tear-down 2022-09-27T16:21:33.3991914Z [==========] 8 tests from 3 test suites ran. (424 ms total) 2022-09-27T16:21:33.3992387Z [ PASSED ] 8 tests. 2022-09-27T16:21:33.3992608Z 2022-09-27T16:21:33.3992812Z  YOU HAVE 1 DISABLED TEST 2022-09-27T16:21:33.3993038Z 2022-09-27T16:21:33.4689694Z  2022-09-27T16:21:33.4719836Z Prepare all required actions 2022-09-27T16:21:33.4720159Z Getting action download info 2022-09-27T16:21:33.7605979Z ##[group]Run ./.github/actions/get-workflow-job-id 2022-09-27T16:21:33.7606209Z with: 2022-09-27T16:21:33.7606537Z github-token: *** 2022-09-27T16:21:33.7606697Z env: 2022-09-27T16:21:33.7606872Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:33.7607063Z ##[endgroup] 2022-09-27T16:21:33.7675814Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767 2022-09-27T16:21:33.7676050Z with: 2022-09-27T16:21:33.7676195Z shell: bash 2022-09-27T16:21:33.7676369Z timeout_minutes: 10 2022-09-27T16:21:33.7676546Z max_attempts: 5 2022-09-27T16:21:33.7676712Z retry_wait_seconds: 30 2022-09-27T16:21:33.7677091Z command: set -eux python3 -m pip install requests==2.26.0 GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}") echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}" 2022-09-27T16:21:33.7677527Z polling_interval_seconds: 1 2022-09-27T16:21:33.7677714Z warning_on_retry: true 2022-09-27T16:21:33.7677903Z continue_on_error: false 2022-09-27T16:21:33.7678075Z env: 2022-09-27T16:21:33.7678234Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:33.7678570Z GITHUB_TOKEN: *** 2022-09-27T16:21:33.7678746Z ##[endgroup] 2022-09-27T16:21:33.8069367Z 2022-09-27T16:21:33.8117074Z + python3 -m pip install requests==2.26.0 2022-09-27T16:21:34.0178471Z Defaulting to user installation because normal site-packages is not writeable 2022-09-27T16:21:34.0351340Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0) 2022-09-27T16:21:34.0478875Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12) 2022-09-27T16:21:34.0501294Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.12) 2022-09-27T16:21:34.0651770Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4) 2022-09-27T16:21:34.0662264Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.9.24) 2022-09-27T16:21:34.2230082Z ++ python3 .github/scripts/get_workflow_job_id.py 3133193930 i-00a9b5b3060f0b9ae 2022-09-27T16:21:39.5712809Z + GHA_WORKFLOW_JOB_ID=8576161248 2022-09-27T16:21:39.5713496Z + echo '::set-output name=job-id::8576161248' 2022-09-27T16:21:39.8161151Z Command completed after 1 attempt(s). 2022-09-27T16:21:39.8161421Z 2022-09-27T16:21:39.8268582Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2022-09-27T16:21:39.8268811Z kill "$MONITOR_SCRIPT_PID" 2022-09-27T16:21:39.8280290Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T16:21:39.8280515Z env: 2022-09-27T16:21:39.8280680Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:39.8280882Z MONITOR_SCRIPT_PID: 21506 2022-09-27T16:21:39.8281069Z ##[endgroup] 2022-09-27T16:21:39.8354349Z Prepare all required actions 2022-09-27T16:21:39.8354772Z Getting action download info 2022-09-27T16:21:39.9927678Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2) 2022-09-27T16:21:40.1881969Z ##[group]Run ./.github/actions/upload-test-artifacts 2022-09-27T16:21:40.1882191Z with: 2022-09-27T16:21:40.1882415Z file-suffix: test-distributed-1-2-linux.2xlarge_8576161248 2022-09-27T16:21:40.1882643Z env: 2022-09-27T16:21:40.1882819Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:40.1882991Z ##[endgroup] 2022-09-27T16:21:40.1906027Z ##[group]Run # Remove any previous test jsons if they exist 2022-09-27T16:21:40.1906315Z # Remove any previous test jsons if they exist 2022-09-27T16:21:40.1906548Z rm -f test-jsons-*.zip 2022-09-27T16:21:40.1906783Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2022-09-27T16:21:40.1918008Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T16:21:40.1918261Z env: 2022-09-27T16:21:40.1918426Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:40.1918686Z FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_8576161248 2022-09-27T16:21:40.1918932Z ##[endgroup] 2022-09-27T16:21:40.2054621Z adding: test/allowlist_for_publicAPI.json (deflated 80%) 2022-09-27T16:21:40.2081623Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2022-09-27T16:21:40.2087489Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2022-09-27T16:21:40.2088685Z adding: test/.pytorch-slow-tests.json (deflated 75%) 2022-09-27T16:21:40.2093896Z adding: test/.pytorch-disabled-tests.json (deflated 85%) 2022-09-27T16:21:40.2113676Z ##[group]Run # Remove any previous test reports if they exist 2022-09-27T16:21:40.2113973Z # Remove any previous test reports if they exist 2022-09-27T16:21:40.2114214Z rm -f test-reports-*.zip 2022-09-27T16:21:40.2114455Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' 2022-09-27T16:21:40.2125599Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T16:21:40.2125828Z env: 2022-09-27T16:21:40.2126016Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:40.2126265Z FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_8576161248 2022-09-27T16:21:40.2126501Z ##[endgroup] 2022-09-27T16:21:40.2256821Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20220927152906.xml (deflated 93%) 2022-09-27T16:21:40.2257506Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220927153034.xml (deflated 42%) 2022-09-27T16:21:40.2258167Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220927153036.xml (deflated 42%) 2022-09-27T16:21:40.2258831Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220927153038.xml (deflated 41%) 2022-09-27T16:21:40.2259490Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220927153042.xml (deflated 41%) 2022-09-27T16:21:40.2260172Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220927153047.xml (deflated 41%) 2022-09-27T16:21:40.2260880Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220927153053.xml (deflated 42%) 2022-09-27T16:21:40.2261586Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220927153058.xml (deflated 41%) 2022-09-27T16:21:40.2262287Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220927153104.xml (deflated 41%) 2022-09-27T16:21:40.2263066Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153110.xml (deflated 44%) 2022-09-27T16:21:40.2263709Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153114.xml (deflated 41%) 2022-09-27T16:21:40.2264432Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153119.xml (deflated 40%) 2022-09-27T16:21:40.2265089Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153123.xml (deflated 41%) 2022-09-27T16:21:40.2265726Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153128.xml (deflated 40%) 2022-09-27T16:21:40.2266353Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153132.xml (deflated 40%) 2022-09-27T16:21:40.2266996Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153136.xml (deflated 41%) 2022-09-27T16:21:40.2267642Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153141.xml (deflated 40%) 2022-09-27T16:21:40.2268285Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153145.xml (deflated 42%) 2022-09-27T16:21:40.2268901Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153147.xml (deflated 40%) 2022-09-27T16:21:40.2269530Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153151.xml (deflated 41%) 2022-09-27T16:21:40.2270172Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153153.xml (deflated 41%) 2022-09-27T16:21:40.2270879Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153158.xml (deflated 41%) 2022-09-27T16:21:40.2271506Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153202.xml (deflated 41%) 2022-09-27T16:21:40.2272150Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153206.xml (deflated 40%) 2022-09-27T16:21:40.2272782Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153211.xml (deflated 40%) 2022-09-27T16:21:40.2273413Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153215.xml (deflated 40%) 2022-09-27T16:21:40.2274042Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153219.xml (deflated 40%) 2022-09-27T16:21:40.2274807Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153223.xml (deflated 40%) 2022-09-27T16:21:40.2275458Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153228.xml (deflated 40%) 2022-09-27T16:21:40.2276098Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153232.xml (deflated 41%) 2022-09-27T16:21:40.2276736Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153236.xml (deflated 40%) 2022-09-27T16:21:40.2277369Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153241.xml (deflated 40%) 2022-09-27T16:21:40.2278068Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153255.xml (deflated 42%) 2022-09-27T16:21:40.2278706Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153257.xml (deflated 40%) 2022-09-27T16:21:40.2279389Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153301.xml (deflated 40%) 2022-09-27T16:21:40.2280014Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153306.xml (deflated 40%) 2022-09-27T16:21:40.2280642Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153310.xml (deflated 41%) 2022-09-27T16:21:40.2281274Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153314.xml (deflated 49%) 2022-09-27T16:21:40.2281911Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153319.xml (deflated 41%) 2022-09-27T16:21:40.2282543Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153323.xml (deflated 41%) 2022-09-27T16:21:40.2283162Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153328.xml (deflated 41%) 2022-09-27T16:21:40.2283806Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153332.xml (deflated 41%) 2022-09-27T16:21:40.2284440Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153337.xml (deflated 41%) 2022-09-27T16:21:40.2285073Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153341.xml (deflated 42%) 2022-09-27T16:21:40.2285695Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153343.xml (deflated 41%) 2022-09-27T16:21:40.2286323Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153348.xml (deflated 41%) 2022-09-27T16:21:40.2286957Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153352.xml (deflated 41%) 2022-09-27T16:21:40.2287589Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153356.xml (deflated 41%) 2022-09-27T16:21:40.2288208Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153401.xml (deflated 40%) 2022-09-27T16:21:40.2288839Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153406.xml (deflated 40%) 2022-09-27T16:21:40.2289473Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153410.xml (deflated 41%) 2022-09-27T16:21:40.2290112Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153414.xml (deflated 40%) 2022-09-27T16:21:40.2290741Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153419.xml (deflated 41%) 2022-09-27T16:21:40.2291373Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153423.xml (deflated 40%) 2022-09-27T16:21:40.2292002Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153428.xml (deflated 40%) 2022-09-27T16:21:40.2292671Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153433.xml (deflated 40%) 2022-09-27T16:21:40.2293305Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153437.xml (deflated 41%) 2022-09-27T16:21:40.2293956Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153441.xml (deflated 40%) 2022-09-27T16:21:40.2294583Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153446.xml (deflated 40%) 2022-09-27T16:21:40.2295216Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153451.xml (deflated 41%) 2022-09-27T16:21:40.2295850Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153455.xml (deflated 40%) 2022-09-27T16:21:40.2296473Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153500.xml (deflated 44%) 2022-09-27T16:21:40.2297105Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153504.xml (deflated 41%) 2022-09-27T16:21:40.2297740Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153508.xml (deflated 41%) 2022-09-27T16:21:40.2298368Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153512.xml (deflated 41%) 2022-09-27T16:21:40.2298999Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153517.xml (deflated 41%) 2022-09-27T16:21:40.2299617Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153521.xml (deflated 40%) 2022-09-27T16:21:40.2300251Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153526.xml (deflated 40%) 2022-09-27T16:21:40.2300896Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153530.xml (deflated 41%) 2022-09-27T16:21:40.2301533Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153535.xml (deflated 41%) 2022-09-27T16:21:40.2302154Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153540.xml (deflated 41%) 2022-09-27T16:21:40.2302786Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220927153544.xml (deflated 40%) 2022-09-27T16:21:40.2303429Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220927153548.xml (deflated 42%) 2022-09-27T16:21:40.2304075Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220927153550.xml (deflated 42%) 2022-09-27T16:21:40.2304715Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220927153555.xml (deflated 42%) 2022-09-27T16:21:40.2305337Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220927153559.xml (deflated 42%) 2022-09-27T16:21:40.2305993Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220927153601.xml (deflated 41%) 2022-09-27T16:21:40.2306657Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220927153605.xml (deflated 41%) 2022-09-27T16:21:40.2307320Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220927153610.xml (deflated 41%) 2022-09-27T16:21:40.2308001Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220927153614.xml (deflated 40%) 2022-09-27T16:21:40.2308665Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153619.xml (deflated 40%) 2022-09-27T16:21:40.2309263Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153623.xml (deflated 39%) 2022-09-27T16:21:40.2309858Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153627.xml (deflated 40%) 2022-09-27T16:21:40.2310434Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153632.xml (deflated 40%) 2022-09-27T16:21:40.2311110Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153636.xml (deflated 39%) 2022-09-27T16:21:40.2311711Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153640.xml (deflated 40%) 2022-09-27T16:21:40.2312312Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153645.xml (deflated 39%) 2022-09-27T16:21:40.2312918Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153649.xml (deflated 39%) 2022-09-27T16:21:40.2313493Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153653.xml (deflated 39%) 2022-09-27T16:21:40.2314078Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153658.xml (deflated 39%) 2022-09-27T16:21:40.2314779Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153702.xml (deflated 40%) 2022-09-27T16:21:40.2315379Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153706.xml (deflated 40%) 2022-09-27T16:21:40.2315957Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153711.xml (deflated 39%) 2022-09-27T16:21:40.2316547Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153715.xml (deflated 39%) 2022-09-27T16:21:40.2317140Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153719.xml (deflated 40%) 2022-09-27T16:21:40.2317727Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153723.xml (deflated 40%) 2022-09-27T16:21:40.2318301Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153728.xml (deflated 40%) 2022-09-27T16:21:40.2318891Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153732.xml (deflated 40%) 2022-09-27T16:21:40.2319486Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153737.xml (deflated 40%) 2022-09-27T16:21:40.2320077Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153741.xml (deflated 39%) 2022-09-27T16:21:40.2320650Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153745.xml (deflated 40%) 2022-09-27T16:21:40.2321237Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153749.xml (deflated 40%) 2022-09-27T16:21:40.2321899Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153754.xml (deflated 40%) 2022-09-27T16:21:40.2322490Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153758.xml (deflated 41%) 2022-09-27T16:21:40.2323112Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153800.xml (deflated 40%) 2022-09-27T16:21:40.2323702Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153804.xml (deflated 40%) 2022-09-27T16:21:40.2324295Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153809.xml (deflated 40%) 2022-09-27T16:21:40.2324886Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153813.xml (deflated 39%) 2022-09-27T16:21:40.2325462Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153817.xml (deflated 40%) 2022-09-27T16:21:40.2326048Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153822.xml (deflated 40%) 2022-09-27T16:21:40.2326636Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153826.xml (deflated 39%) 2022-09-27T16:21:40.2327240Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153830.xml (deflated 40%) 2022-09-27T16:21:40.2327829Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153835.xml (deflated 39%) 2022-09-27T16:21:40.2328413Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153840.xml (deflated 40%) 2022-09-27T16:21:40.2329007Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153844.xml (deflated 40%) 2022-09-27T16:21:40.2329592Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153848.xml (deflated 40%) 2022-09-27T16:21:40.2330182Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153853.xml (deflated 40%) 2022-09-27T16:21:40.2330756Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153857.xml (deflated 40%) 2022-09-27T16:21:40.2331341Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153902.xml (deflated 40%) 2022-09-27T16:21:40.2331929Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153906.xml (deflated 39%) 2022-09-27T16:21:40.2332522Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153910.xml (deflated 40%) 2022-09-27T16:21:40.2333095Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153915.xml (deflated 40%) 2022-09-27T16:21:40.2333687Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153919.xml (deflated 40%) 2022-09-27T16:21:40.2334270Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153923.xml (deflated 40%) 2022-09-27T16:21:40.2334856Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153928.xml (deflated 40%) 2022-09-27T16:21:40.2335438Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153932.xml (deflated 40%) 2022-09-27T16:21:40.2336048Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153937.xml (deflated 39%) 2022-09-27T16:21:40.2336635Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153941.xml (deflated 39%) 2022-09-27T16:21:40.2337263Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153945.xml (deflated 40%) 2022-09-27T16:21:40.2337857Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153950.xml (deflated 40%) 2022-09-27T16:21:40.2338434Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153954.xml (deflated 40%) 2022-09-27T16:21:40.2339021Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927153958.xml (deflated 40%) 2022-09-27T16:21:40.2339615Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927154003.xml (deflated 40%) 2022-09-27T16:21:40.2340200Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220927154007.xml (deflated 40%) 2022-09-27T16:21:40.2340824Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeParameterServerTest-20220927154012.xml (deflated 43%) 2022-09-27T16:21:40.2341528Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeReinforcementLearningRpcTest-20220927154016.xml (deflated 43%) 2022-09-27T16:21:40.2342206Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154021.xml (deflated 41%) 2022-09-27T16:21:40.2342854Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154024.xml (deflated 41%) 2022-09-27T16:21:40.2343480Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154029.xml (deflated 40%) 2022-09-27T16:21:40.2344117Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154033.xml (deflated 40%) 2022-09-27T16:21:40.2344785Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154037.xml (deflated 40%) 2022-09-27T16:21:40.2345419Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154041.xml (deflated 40%) 2022-09-27T16:21:40.2346050Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154045.xml (deflated 41%) 2022-09-27T16:21:40.2346675Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154049.xml (deflated 41%) 2022-09-27T16:21:40.2347311Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154053.xml (deflated 41%) 2022-09-27T16:21:40.2347943Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154057.xml (deflated 41%) 2022-09-27T16:21:40.2348574Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154101.xml (deflated 41%) 2022-09-27T16:21:40.2349193Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154105.xml (deflated 41%) 2022-09-27T16:21:40.2349827Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220927154109.xml (deflated 41%) 2022-09-27T16:21:40.2350434Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154113.xml (deflated 40%) 2022-09-27T16:21:40.2351142Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154117.xml (deflated 40%) 2022-09-27T16:21:40.2351783Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154122.xml (deflated 40%) 2022-09-27T16:21:40.2352356Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154126.xml (deflated 40%) 2022-09-27T16:21:40.2352944Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154130.xml (deflated 40%) 2022-09-27T16:21:40.2353525Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154143.xml (deflated 40%) 2022-09-27T16:21:40.2354106Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154148.xml (deflated 40%) 2022-09-27T16:21:40.2354852Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154152.xml (deflated 40%) 2022-09-27T16:21:40.2355484Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154156.xml (deflated 40%) 2022-09-27T16:21:40.2356066Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154201.xml (deflated 40%) 2022-09-27T16:21:40.2356652Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154205.xml (deflated 40%) 2022-09-27T16:21:40.2357409Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154209.xml (deflated 40%) 2022-09-27T16:21:40.2358248Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154213.xml (deflated 39%) 2022-09-27T16:21:40.2358870Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154218.xml (deflated 39%) 2022-09-27T16:21:40.2359459Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154222.xml (deflated 40%) 2022-09-27T16:21:40.2360053Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154226.xml (deflated 39%) 2022-09-27T16:21:40.2360645Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154230.xml (deflated 40%) 2022-09-27T16:21:40.2361226Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154235.xml (deflated 40%) 2022-09-27T16:21:40.2361807Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154239.xml (deflated 39%) 2022-09-27T16:21:40.2362385Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154243.xml (deflated 40%) 2022-09-27T16:21:40.2362998Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154248.xml (deflated 40%) 2022-09-27T16:21:40.2363617Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154252.xml (deflated 40%) 2022-09-27T16:21:40.2364222Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154256.xml (deflated 41%) 2022-09-27T16:21:40.2364868Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154258.xml (deflated 40%) 2022-09-27T16:21:40.2365456Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154303.xml (deflated 39%) 2022-09-27T16:21:40.2366157Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154307.xml (deflated 39%) 2022-09-27T16:21:40.2366810Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154311.xml (deflated 40%) 2022-09-27T16:21:40.2367511Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154316.xml (deflated 39%) 2022-09-27T16:21:40.2368118Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154320.xml (deflated 40%) 2022-09-27T16:21:40.2368732Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154324.xml (deflated 39%) 2022-09-27T16:21:40.2369351Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154329.xml (deflated 39%) 2022-09-27T16:21:40.2369961Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154333.xml (deflated 39%) 2022-09-27T16:21:40.2370598Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154338.xml (deflated 39%) 2022-09-27T16:21:40.2371190Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154343.xml (deflated 40%) 2022-09-27T16:21:40.2371805Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154348.xml (deflated 39%) 2022-09-27T16:21:40.2372427Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154352.xml (deflated 39%) 2022-09-27T16:21:40.2373041Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154356.xml (deflated 40%) 2022-09-27T16:21:40.2373647Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154400.xml (deflated 40%) 2022-09-27T16:21:40.2374265Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154405.xml (deflated 42%) 2022-09-27T16:21:40.2374908Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154409.xml (deflated 39%) 2022-09-27T16:21:40.2375515Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154413.xml (deflated 40%) 2022-09-27T16:21:40.2376116Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154417.xml (deflated 40%) 2022-09-27T16:21:40.2376742Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154422.xml (deflated 40%) 2022-09-27T16:21:40.2377356Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154426.xml (deflated 40%) 2022-09-27T16:21:40.2377961Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154431.xml (deflated 40%) 2022-09-27T16:21:40.2378562Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154435.xml (deflated 39%) 2022-09-27T16:21:40.2379197Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154439.xml (deflated 39%) 2022-09-27T16:21:40.2379809Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154444.xml (deflated 39%) 2022-09-27T16:21:40.2380421Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154448.xml (deflated 39%) 2022-09-27T16:21:40.2381011Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154452.xml (deflated 40%) 2022-09-27T16:21:40.2381684Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154457.xml (deflated 40%) 2022-09-27T16:21:40.2382344Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154502.xml (deflated 44%) 2022-09-27T16:21:40.2382973Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154508.xml (deflated 39%) 2022-09-27T16:21:40.2383606Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154512.xml (deflated 40%) 2022-09-27T16:21:40.2384217Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154517.xml (deflated 40%) 2022-09-27T16:21:40.2384834Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154521.xml (deflated 40%) 2022-09-27T16:21:40.2385445Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154525.xml (deflated 40%) 2022-09-27T16:21:40.2386077Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154529.xml (deflated 40%) 2022-09-27T16:21:40.2386690Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154533.xml (deflated 40%) 2022-09-27T16:21:40.2387304Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154537.xml (deflated 40%) 2022-09-27T16:21:40.2387924Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154542.xml (deflated 40%) 2022-09-27T16:21:40.2388557Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154546.xml (deflated 40%) 2022-09-27T16:21:40.2389161Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154550.xml (deflated 41%) 2022-09-27T16:21:40.2389771Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154552.xml (deflated 40%) 2022-09-27T16:21:40.2390411Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154557.xml (deflated 41%) 2022-09-27T16:21:40.2391115Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154601.xml (deflated 39%) 2022-09-27T16:21:40.2391712Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154605.xml (deflated 40%) 2022-09-27T16:21:40.2392357Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154610.xml (deflated 40%) 2022-09-27T16:21:40.2392982Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154614.xml (deflated 40%) 2022-09-27T16:21:40.2393596Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154619.xml (deflated 40%) 2022-09-27T16:21:40.2394216Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154623.xml (deflated 40%) 2022-09-27T16:21:40.2394956Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154627.xml (deflated 40%) 2022-09-27T16:21:40.2395537Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154632.xml (deflated 39%) 2022-09-27T16:21:40.2396131Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154636.xml (deflated 39%) 2022-09-27T16:21:40.2397065Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154640.xml (deflated 39%) 2022-09-27T16:21:40.2397674Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154644.xml (deflated 40%) 2022-09-27T16:21:40.2398346Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154649.xml (deflated 39%) 2022-09-27T16:21:40.2398970Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154653.xml (deflated 40%) 2022-09-27T16:21:40.2399614Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154657.xml (deflated 40%) 2022-09-27T16:21:40.2400211Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154702.xml (deflated 40%) 2022-09-27T16:21:40.2400827Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154706.xml (deflated 40%) 2022-09-27T16:21:40.2401472Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154710.xml (deflated 40%) 2022-09-27T16:21:40.2402085Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154715.xml (deflated 40%) 2022-09-27T16:21:40.2402686Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154719.xml (deflated 40%) 2022-09-27T16:21:40.2403314Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154724.xml (deflated 40%) 2022-09-27T16:21:40.2403950Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154728.xml (deflated 40%) 2022-09-27T16:21:40.2404565Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154732.xml (deflated 41%) 2022-09-27T16:21:40.2405165Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154736.xml (deflated 39%) 2022-09-27T16:21:40.2405798Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154741.xml (deflated 40%) 2022-09-27T16:21:40.2406445Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154745.xml (deflated 40%) 2022-09-27T16:21:40.2407050Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154749.xml (deflated 39%) 2022-09-27T16:21:40.2407641Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154753.xml (deflated 40%) 2022-09-27T16:21:40.2408283Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154758.xml (deflated 41%) 2022-09-27T16:21:40.2408900Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154802.xml (deflated 40%) 2022-09-27T16:21:40.2409515Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154806.xml (deflated 40%) 2022-09-27T16:21:40.2410112Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154810.xml (deflated 41%) 2022-09-27T16:21:40.2410747Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154815.xml (deflated 40%) 2022-09-27T16:21:40.2411358Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154819.xml (deflated 40%) 2022-09-27T16:21:40.2412011Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154823.xml (deflated 40%) 2022-09-27T16:21:40.2412633Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154828.xml (deflated 40%) 2022-09-27T16:21:40.2413279Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154832.xml (deflated 39%) 2022-09-27T16:21:40.2413921Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154836.xml (deflated 39%) 2022-09-27T16:21:40.2414546Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154840.xml (deflated 40%) 2022-09-27T16:21:40.2415162Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154848.xml (deflated 39%) 2022-09-27T16:21:40.2415768Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154855.xml (deflated 40%) 2022-09-27T16:21:40.2416410Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154859.xml (deflated 39%) 2022-09-27T16:21:40.2417035Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154904.xml (deflated 39%) 2022-09-27T16:21:40.2417656Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154908.xml (deflated 39%) 2022-09-27T16:21:40.2418273Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154912.xml (deflated 40%) 2022-09-27T16:21:40.2418890Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154919.xml (deflated 40%) 2022-09-27T16:21:40.2419499Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154927.xml (deflated 40%) 2022-09-27T16:21:40.2420145Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154931.xml (deflated 40%) 2022-09-27T16:21:40.2420741Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154936.xml (deflated 39%) 2022-09-27T16:21:40.2421364Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154940.xml (deflated 40%) 2022-09-27T16:21:40.2422013Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154944.xml (deflated 40%) 2022-09-27T16:21:40.2422626Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154948.xml (deflated 40%) 2022-09-27T16:21:40.2423232Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154953.xml (deflated 40%) 2022-09-27T16:21:40.2423844Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927154957.xml (deflated 40%) 2022-09-27T16:21:40.2424487Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155001.xml (deflated 40%) 2022-09-27T16:21:40.2425098Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155007.xml (deflated 39%) 2022-09-27T16:21:40.2425696Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155014.xml (deflated 40%) 2022-09-27T16:21:40.2426344Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155018.xml (deflated 40%) 2022-09-27T16:21:40.2426958Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155022.xml (deflated 40%) 2022-09-27T16:21:40.2427616Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155027.xml (deflated 40%) 2022-09-27T16:21:40.2428279Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155031.xml (deflated 40%) 2022-09-27T16:21:40.2428898Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155035.xml (deflated 40%) 2022-09-27T16:21:40.2429506Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155040.xml (deflated 40%) 2022-09-27T16:21:40.2430142Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155044.xml (deflated 40%) 2022-09-27T16:21:40.2430819Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155048.xml (deflated 40%) 2022-09-27T16:21:40.2431443Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155053.xml (deflated 40%) 2022-09-27T16:21:40.2432088Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155057.xml (deflated 40%) 2022-09-27T16:21:40.2432764Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155101.xml (deflated 40%) 2022-09-27T16:21:40.2433375Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155106.xml (deflated 40%) 2022-09-27T16:21:40.2434019Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155110.xml (deflated 40%) 2022-09-27T16:21:40.2434727Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155114.xml (deflated 41%) 2022-09-27T16:21:40.2435355Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155119.xml (deflated 41%) 2022-09-27T16:21:40.2435976Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155123.xml (deflated 41%) 2022-09-27T16:21:40.2436604Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155127.xml (deflated 40%) 2022-09-27T16:21:40.2437217Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155132.xml (deflated 40%) 2022-09-27T16:21:40.2437832Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155136.xml (deflated 41%) 2022-09-27T16:21:40.2438457Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155140.xml (deflated 40%) 2022-09-27T16:21:40.2439098Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155144.xml (deflated 40%) 2022-09-27T16:21:40.2439710Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155149.xml (deflated 40%) 2022-09-27T16:21:40.2440345Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155153.xml (deflated 40%) 2022-09-27T16:21:40.2440960Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155157.xml (deflated 40%) 2022-09-27T16:21:40.2441566Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155202.xml (deflated 40%) 2022-09-27T16:21:40.2442195Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155206.xml (deflated 40%) 2022-09-27T16:21:40.2442889Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155210.xml (deflated 40%) 2022-09-27T16:21:40.2443507Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155215.xml (deflated 40%) 2022-09-27T16:21:40.2444201Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155219.xml (deflated 40%) 2022-09-27T16:21:40.2444797Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155223.xml (deflated 40%) 2022-09-27T16:21:40.2445408Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155228.xml (deflated 40%) 2022-09-27T16:21:40.2446022Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155232.xml (deflated 40%) 2022-09-27T16:21:40.2446675Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155237.xml (deflated 40%) 2022-09-27T16:21:40.2447278Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155243.xml (deflated 40%) 2022-09-27T16:21:40.2447897Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155247.xml (deflated 40%) 2022-09-27T16:21:40.2448502Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155251.xml (deflated 40%) 2022-09-27T16:21:40.2449143Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155302.xml (deflated 39%) 2022-09-27T16:21:40.2449739Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155306.xml (deflated 39%) 2022-09-27T16:21:40.2450353Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155311.xml (deflated 40%) 2022-09-27T16:21:40.2450973Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155315.xml (deflated 40%) 2022-09-27T16:21:40.2451623Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155319.xml (deflated 40%) 2022-09-27T16:21:40.2452232Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155324.xml (deflated 40%) 2022-09-27T16:21:40.2452830Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155328.xml (deflated 40%) 2022-09-27T16:21:40.2453439Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155332.xml (deflated 40%) 2022-09-27T16:21:40.2454054Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155336.xml (deflated 40%) 2022-09-27T16:21:40.2454664Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155341.xml (deflated 40%) 2022-09-27T16:21:40.2455270Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155345.xml (deflated 39%) 2022-09-27T16:21:40.2455917Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155349.xml (deflated 40%) 2022-09-27T16:21:40.2456535Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155354.xml (deflated 40%) 2022-09-27T16:21:40.2457149Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155400.xml (deflated 40%) 2022-09-27T16:21:40.2457785Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155404.xml (deflated 40%) 2022-09-27T16:21:40.2458421Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155409.xml (deflated 39%) 2022-09-27T16:21:40.2459101Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155413.xml (deflated 39%) 2022-09-27T16:21:40.2459714Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155417.xml (deflated 40%) 2022-09-27T16:21:40.2460344Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155423.xml (deflated 40%) 2022-09-27T16:21:40.2460951Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155427.xml (deflated 40%) 2022-09-27T16:21:40.2461563Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155431.xml (deflated 40%) 2022-09-27T16:21:40.2462206Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155435.xml (deflated 40%) 2022-09-27T16:21:40.2462806Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155440.xml (deflated 40%) 2022-09-27T16:21:40.2463427Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155444.xml (deflated 41%) 2022-09-27T16:21:40.2464046Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155449.xml (deflated 40%) 2022-09-27T16:21:40.2464689Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155453.xml (deflated 41%) 2022-09-27T16:21:40.2465293Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155457.xml (deflated 41%) 2022-09-27T16:21:40.2465909Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155502.xml (deflated 40%) 2022-09-27T16:21:40.2466548Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155506.xml (deflated 40%) 2022-09-27T16:21:40.2467190Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155510.xml (deflated 40%) 2022-09-27T16:21:40.2467793Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155514.xml (deflated 40%) 2022-09-27T16:21:40.2468416Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155519.xml (deflated 39%) 2022-09-27T16:21:40.2469046Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155523.xml (deflated 40%) 2022-09-27T16:21:40.2469672Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155527.xml (deflated 40%) 2022-09-27T16:21:40.2470273Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155532.xml (deflated 40%) 2022-09-27T16:21:40.2470959Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155536.xml (deflated 40%) 2022-09-27T16:21:40.2471578Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155540.xml (deflated 39%) 2022-09-27T16:21:40.2472230Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155544.xml (deflated 39%) 2022-09-27T16:21:40.2472850Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155549.xml (deflated 40%) 2022-09-27T16:21:40.2473503Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155553.xml (deflated 40%) 2022-09-27T16:21:40.2474151Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155558.xml (deflated 40%) 2022-09-27T16:21:40.2474882Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155602.xml (deflated 40%) 2022-09-27T16:21:40.2475524Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155606.xml (deflated 39%) 2022-09-27T16:21:40.2476127Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155611.xml (deflated 40%) 2022-09-27T16:21:40.2476750Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155615.xml (deflated 40%) 2022-09-27T16:21:40.2477378Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155619.xml (deflated 40%) 2022-09-27T16:21:40.2477986Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155624.xml (deflated 40%) 2022-09-27T16:21:40.2478588Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155628.xml (deflated 40%) 2022-09-27T16:21:40.2479249Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155633.xml (deflated 39%) 2022-09-27T16:21:40.2479937Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155637.xml (deflated 39%) 2022-09-27T16:21:40.2480545Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155642.xml (deflated 40%) 2022-09-27T16:21:40.2481167Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155646.xml (deflated 40%) 2022-09-27T16:21:40.2481759Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155652.xml (deflated 40%) 2022-09-27T16:21:40.2482377Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155656.xml (deflated 40%) 2022-09-27T16:21:40.2482995Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155701.xml (deflated 39%) 2022-09-27T16:21:40.2483620Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155705.xml (deflated 40%) 2022-09-27T16:21:40.2484211Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220927155709.xml (deflated 40%) 2022-09-27T16:21:40.2484918Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155713.xml (deflated 43%) 2022-09-27T16:21:40.2485717Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155717.xml (deflated 43%) 2022-09-27T16:21:40.2486516Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155722.xml (deflated 44%) 2022-09-27T16:21:40.2487328Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155726.xml (deflated 44%) 2022-09-27T16:21:40.2488086Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155730.xml (deflated 44%) 2022-09-27T16:21:40.2488954Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155735.xml (deflated 44%) 2022-09-27T16:21:40.2489754Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155739.xml (deflated 43%) 2022-09-27T16:21:40.2490622Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155744.xml (deflated 43%) 2022-09-27T16:21:40.2491390Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155748.xml (deflated 44%) 2022-09-27T16:21:40.2492201Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155752.xml (deflated 44%) 2022-09-27T16:21:40.2492986Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155757.xml (deflated 44%) 2022-09-27T16:21:40.2493796Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155801.xml (deflated 44%) 2022-09-27T16:21:40.2494571Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155806.xml (deflated 44%) 2022-09-27T16:21:40.2495334Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155810.xml (deflated 44%) 2022-09-27T16:21:40.2496159Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155815.xml (deflated 44%) 2022-09-27T16:21:40.2496937Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155819.xml (deflated 44%) 2022-09-27T16:21:40.2497743Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155824.xml (deflated 43%) 2022-09-27T16:21:40.2498519Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155828.xml (deflated 43%) 2022-09-27T16:21:40.2499315Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155833.xml (deflated 43%) 2022-09-27T16:21:40.2500124Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155837.xml (deflated 43%) 2022-09-27T16:21:40.2500907Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155842.xml (deflated 43%) 2022-09-27T16:21:40.2501736Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155846.xml (deflated 43%) 2022-09-27T16:21:40.2502504Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155851.xml (deflated 43%) 2022-09-27T16:21:40.2503312Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155855.xml (deflated 44%) 2022-09-27T16:21:40.2504090Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155900.xml (deflated 44%) 2022-09-27T16:21:40.2504900Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155905.xml (deflated 43%) 2022-09-27T16:21:40.2505666Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155910.xml (deflated 44%) 2022-09-27T16:21:40.2506532Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155914.xml (deflated 44%) 2022-09-27T16:21:40.2507375Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155919.xml (deflated 44%) 2022-09-27T16:21:40.2508156Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155923.xml (deflated 44%) 2022-09-27T16:21:40.2508953Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220927155928.xml (deflated 44%) 2022-09-27T16:21:40.2509704Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155932.xml (deflated 43%) 2022-09-27T16:21:40.2510459Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155936.xml (deflated 43%) 2022-09-27T16:21:40.2511453Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155941.xml (deflated 43%) 2022-09-27T16:21:40.2512149Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155945.xml (deflated 43%) 2022-09-27T16:21:40.2512822Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155947.xml (deflated 43%) 2022-09-27T16:21:40.2513501Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155951.xml (deflated 43%) 2022-09-27T16:21:40.2514181Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155955.xml (deflated 43%) 2022-09-27T16:21:40.2514984Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927155959.xml (deflated 42%) 2022-09-27T16:21:40.2515662Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160004.xml (deflated 43%) 2022-09-27T16:21:40.2516341Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160008.xml (deflated 43%) 2022-09-27T16:21:40.2517016Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160012.xml (deflated 43%) 2022-09-27T16:21:40.2517700Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160017.xml (deflated 43%) 2022-09-27T16:21:40.2518369Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160021.xml (deflated 43%) 2022-09-27T16:21:40.2519038Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160026.xml (deflated 43%) 2022-09-27T16:21:40.2519708Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160030.xml (deflated 43%) 2022-09-27T16:21:40.2520388Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160035.xml (deflated 43%) 2022-09-27T16:21:40.2521071Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160039.xml (deflated 43%) 2022-09-27T16:21:40.2521738Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160044.xml (deflated 43%) 2022-09-27T16:21:40.2522486Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160048.xml (deflated 43%) 2022-09-27T16:21:40.2523218Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160052.xml (deflated 43%) 2022-09-27T16:21:40.2523896Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160057.xml (deflated 43%) 2022-09-27T16:21:40.2524560Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160101.xml (deflated 43%) 2022-09-27T16:21:40.2525237Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160106.xml (deflated 43%) 2022-09-27T16:21:40.2525914Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160110.xml (deflated 43%) 2022-09-27T16:21:40.2526591Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160116.xml (deflated 43%) 2022-09-27T16:21:40.2527259Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160123.xml (deflated 43%) 2022-09-27T16:21:40.2527927Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160140.xml (deflated 43%) 2022-09-27T16:21:40.2528593Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160144.xml (deflated 44%) 2022-09-27T16:21:40.2529269Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160149.xml (deflated 43%) 2022-09-27T16:21:40.2529942Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160153.xml (deflated 44%) 2022-09-27T16:21:40.2530600Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160157.xml (deflated 43%) 2022-09-27T16:21:40.2531267Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160202.xml (deflated 43%) 2022-09-27T16:21:40.2531935Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160206.xml (deflated 42%) 2022-09-27T16:21:40.2532609Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160210.xml (deflated 43%) 2022-09-27T16:21:40.2533268Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160215.xml (deflated 44%) 2022-09-27T16:21:40.2533941Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160219.xml (deflated 44%) 2022-09-27T16:21:40.2534617Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160223.xml (deflated 43%) 2022-09-27T16:21:40.2535286Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160228.xml (deflated 43%) 2022-09-27T16:21:40.2535946Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220927160232.xml (deflated 43%) 2022-09-27T16:21:40.2536652Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220927160236.xml (deflated 43%) 2022-09-27T16:21:40.2537409Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220927160240.xml (deflated 42%) 2022-09-27T16:21:40.2538132Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220927160245.xml (deflated 42%) 2022-09-27T16:21:40.2538872Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20220927160250.xml (deflated 42%) 2022-09-27T16:21:40.2539554Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220927160252.xml (deflated 42%) 2022-09-27T16:21:40.2540241Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220927160256.xml (deflated 42%) 2022-09-27T16:21:40.2540921Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220927160300.xml (deflated 42%) 2022-09-27T16:21:40.2541604Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220927160304.xml (deflated 42%) 2022-09-27T16:21:40.2542263Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220927160308.xml (deflated 42%) 2022-09-27T16:21:40.2542941Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220927160312.xml (deflated 42%) 2022-09-27T16:21:40.2543615Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220927160315.xml (deflated 42%) 2022-09-27T16:21:40.2544271Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20220927160319.xml (deflated 41%) 2022-09-27T16:21:40.2544904Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160323.xml (deflated 40%) 2022-09-27T16:21:40.2545559Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160327.xml (deflated 41%) 2022-09-27T16:21:40.2546219Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160331.xml (deflated 41%) 2022-09-27T16:21:40.2546925Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160334.xml (deflated 40%) 2022-09-27T16:21:40.2547579Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160338.xml (deflated 41%) 2022-09-27T16:21:40.2548235Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160340.xml (deflated 41%) 2022-09-27T16:21:40.2548886Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160342.xml (deflated 41%) 2022-09-27T16:21:40.2549540Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220927160343.xml (deflated 41%) 2022-09-27T16:21:40.2550239Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160345.xml (deflated 43%) 2022-09-27T16:21:40.2551053Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160349.xml (deflated 43%) 2022-09-27T16:21:40.2551773Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160353.xml (deflated 44%) 2022-09-27T16:21:40.2552538Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160357.xml (deflated 44%) 2022-09-27T16:21:40.2553270Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160401.xml (deflated 44%) 2022-09-27T16:21:40.2554042Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160405.xml (deflated 44%) 2022-09-27T16:21:40.2554891Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160409.xml (deflated 44%) 2022-09-27T16:21:40.2555611Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160413.xml (deflated 44%) 2022-09-27T16:21:40.2556343Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160417.xml (deflated 44%) 2022-09-27T16:21:40.2557072Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160420.xml (deflated 45%) 2022-09-27T16:21:40.2557805Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160424.xml (deflated 44%) 2022-09-27T16:21:40.2558514Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160428.xml (deflated 44%) 2022-09-27T16:21:40.2559240Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160432.xml (deflated 44%) 2022-09-27T16:21:40.2559972Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160436.xml (deflated 44%) 2022-09-27T16:21:40.2560698Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160440.xml (deflated 44%) 2022-09-27T16:21:40.2561425Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160444.xml (deflated 44%) 2022-09-27T16:21:40.2562139Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160448.xml (deflated 43%) 2022-09-27T16:21:40.2562865Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160450.xml (deflated 44%) 2022-09-27T16:21:40.2563591Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160454.xml (deflated 43%) 2022-09-27T16:21:40.2564319Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160458.xml (deflated 43%) 2022-09-27T16:21:40.2565034Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160502.xml (deflated 43%) 2022-09-27T16:21:40.2565758Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160506.xml (deflated 44%) 2022-09-27T16:21:40.2566487Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160510.xml (deflated 44%) 2022-09-27T16:21:40.2567214Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160514.xml (deflated 44%) 2022-09-27T16:21:40.2567937Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160518.xml (deflated 43%) 2022-09-27T16:21:40.2568707Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160520.xml (deflated 44%) 2022-09-27T16:21:40.2569500Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160524.xml (deflated 44%) 2022-09-27T16:21:40.2570223Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160528.xml (deflated 43%) 2022-09-27T16:21:40.2570952Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160532.xml (deflated 43%) 2022-09-27T16:21:40.2571666Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160536.xml (deflated 43%) 2022-09-27T16:21:40.2572393Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160540.xml (deflated 44%) 2022-09-27T16:21:40.2573180Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160544.xml (deflated 43%) 2022-09-27T16:21:40.2573909Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160548.xml (deflated 44%) 2022-09-27T16:21:40.2574639Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160552.xml (deflated 44%) 2022-09-27T16:21:40.2575363Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160556.xml (deflated 43%) 2022-09-27T16:21:40.2576085Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160600.xml (deflated 44%) 2022-09-27T16:21:40.2576813Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160604.xml (deflated 44%) 2022-09-27T16:21:40.2577544Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160608.xml (deflated 43%) 2022-09-27T16:21:40.2578271Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160612.xml (deflated 43%) 2022-09-27T16:21:40.2578994Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160616.xml (deflated 43%) 2022-09-27T16:21:40.2579709Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160620.xml (deflated 44%) 2022-09-27T16:21:40.2580438Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160624.xml (deflated 43%) 2022-09-27T16:21:40.2581169Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160628.xml (deflated 44%) 2022-09-27T16:21:40.2581892Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160632.xml (deflated 43%) 2022-09-27T16:21:40.2582608Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160636.xml (deflated 43%) 2022-09-27T16:21:40.2583342Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160640.xml (deflated 44%) 2022-09-27T16:21:40.2584100Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160644.xml (deflated 44%) 2022-09-27T16:21:40.2584823Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160648.xml (deflated 44%) 2022-09-27T16:21:40.2585623Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160652.xml (deflated 44%) 2022-09-27T16:21:40.2586336Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160656.xml (deflated 44%) 2022-09-27T16:21:40.2587068Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160700.xml (deflated 44%) 2022-09-27T16:21:40.2587792Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160704.xml (deflated 44%) 2022-09-27T16:21:40.2588519Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160708.xml (deflated 44%) 2022-09-27T16:21:40.2589242Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160712.xml (deflated 44%) 2022-09-27T16:21:40.2589971Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160716.xml (deflated 43%) 2022-09-27T16:21:40.2590763Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160720.xml (deflated 44%) 2022-09-27T16:21:40.2591497Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160724.xml (deflated 43%) 2022-09-27T16:21:40.2592229Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160728.xml (deflated 43%) 2022-09-27T16:21:40.2592946Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160732.xml (deflated 44%) 2022-09-27T16:21:40.2593677Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160736.xml (deflated 44%) 2022-09-27T16:21:40.2594405Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160740.xml (deflated 44%) 2022-09-27T16:21:40.2595253Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160744.xml (deflated 44%) 2022-09-27T16:21:40.2595968Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160748.xml (deflated 43%) 2022-09-27T16:21:40.2596704Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160752.xml (deflated 43%) 2022-09-27T16:21:40.2597427Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160756.xml (deflated 43%) 2022-09-27T16:21:40.2598157Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160800.xml (deflated 44%) 2022-09-27T16:21:40.2598886Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160804.xml (deflated 44%) 2022-09-27T16:21:40.2599662Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160807.xml (deflated 44%) 2022-09-27T16:21:40.2600401Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160811.xml (deflated 43%) 2022-09-27T16:21:40.2601175Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160815.xml (deflated 43%) 2022-09-27T16:21:40.2601907Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160819.xml (deflated 43%) 2022-09-27T16:21:40.2602618Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160824.xml (deflated 43%) 2022-09-27T16:21:40.2603340Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160827.xml (deflated 43%) 2022-09-27T16:21:40.2604070Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160831.xml (deflated 43%) 2022-09-27T16:21:40.2604799Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160835.xml (deflated 43%) 2022-09-27T16:21:40.2605522Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160837.xml (deflated 43%) 2022-09-27T16:21:40.2606232Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160841.xml (deflated 43%) 2022-09-27T16:21:40.2606953Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160845.xml (deflated 43%) 2022-09-27T16:21:40.2607680Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160849.xml (deflated 43%) 2022-09-27T16:21:40.2608407Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160853.xml (deflated 43%) 2022-09-27T16:21:40.2609116Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160857.xml (deflated 43%) 2022-09-27T16:21:40.2609835Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160901.xml (deflated 43%) 2022-09-27T16:21:40.2610562Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160905.xml (deflated 43%) 2022-09-27T16:21:40.2611290Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160909.xml (deflated 43%) 2022-09-27T16:21:40.2612015Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160913.xml (deflated 43%) 2022-09-27T16:21:40.2612731Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160917.xml (deflated 44%) 2022-09-27T16:21:40.2613452Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220927160921.xml (deflated 44%) 2022-09-27T16:21:40.2614196Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220927160925.xml (deflated 44%) 2022-09-27T16:21:40.2614951Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220927160929.xml (deflated 44%) 2022-09-27T16:21:40.2615728Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220927160933.xml (deflated 44%) 2022-09-27T16:21:40.2616421Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20220927160937.xml (deflated 95%) 2022-09-27T16:21:40.2616967Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161130.xml (deflated 40%) 2022-09-27T16:21:40.2617466Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161131.xml (deflated 41%) 2022-09-27T16:21:40.2617951Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161132.xml (deflated 40%) 2022-09-27T16:21:40.2618448Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161134.xml (deflated 40%) 2022-09-27T16:21:40.2618944Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161135.xml (deflated 39%) 2022-09-27T16:21:40.2619437Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161137.xml (deflated 40%) 2022-09-27T16:21:40.2619927Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161138.xml (deflated 40%) 2022-09-27T16:21:40.2620425Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161139.xml (deflated 39%) 2022-09-27T16:21:40.2620923Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161141.xml (deflated 39%) 2022-09-27T16:21:40.2621420Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161142.xml (deflated 39%) 2022-09-27T16:21:40.2621903Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161144.xml (deflated 39%) 2022-09-27T16:21:40.2622401Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161145.xml (deflated 40%) 2022-09-27T16:21:40.2622890Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161147.xml (deflated 39%) 2022-09-27T16:21:40.2623385Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161148.xml (deflated 39%) 2022-09-27T16:21:40.2623869Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161149.xml (deflated 39%) 2022-09-27T16:21:40.2624364Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161151.xml (deflated 39%) 2022-09-27T16:21:40.2624853Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161152.xml (deflated 39%) 2022-09-27T16:21:40.2625345Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161154.xml (deflated 40%) 2022-09-27T16:21:40.2625831Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927161155.xml (deflated 40%) 2022-09-27T16:21:40.2626342Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161157.xml (deflated 39%) 2022-09-27T16:21:40.2626869Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161200.xml (deflated 40%) 2022-09-27T16:21:40.2627389Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161203.xml (deflated 40%) 2022-09-27T16:21:40.2627892Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161207.xml (deflated 40%) 2022-09-27T16:21:40.2628409Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161210.xml (deflated 40%) 2022-09-27T16:21:40.2628922Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161213.xml (deflated 40%) 2022-09-27T16:21:40.2629464Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927161217.xml (deflated 40%) 2022-09-27T16:21:40.2630048Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161220.xml (deflated 42%) 2022-09-27T16:21:40.2630853Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161221.xml (deflated 42%) 2022-09-27T16:21:40.2631732Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161223.xml (deflated 42%) 2022-09-27T16:21:40.2632675Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161224.xml (deflated 41%) 2022-09-27T16:21:40.2633502Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161226.xml (deflated 42%) 2022-09-27T16:21:40.2634119Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161227.xml (deflated 42%) 2022-09-27T16:21:40.2634825Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161229.xml (deflated 42%) 2022-09-27T16:21:40.2635441Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161230.xml (deflated 42%) 2022-09-27T16:21:40.2636057Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161231.xml (deflated 42%) 2022-09-27T16:21:40.2636664Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161233.xml (deflated 44%) 2022-09-27T16:21:40.2637252Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161236.xml (deflated 44%) 2022-09-27T16:21:40.2637850Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161239.xml (deflated 43%) 2022-09-27T16:21:40.2638457Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161243.xml (deflated 42%) 2022-09-27T16:21:40.2639059Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161246.xml (deflated 45%) 2022-09-27T16:21:40.2639650Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161249.xml (deflated 45%) 2022-09-27T16:21:40.2640247Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161253.xml (deflated 46%) 2022-09-27T16:21:40.2640842Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161256.xml (deflated 46%) 2022-09-27T16:21:40.2641444Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161259.xml (deflated 44%) 2022-09-27T16:21:40.2642031Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161303.xml (deflated 45%) 2022-09-27T16:21:40.2642637Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161306.xml (deflated 45%) 2022-09-27T16:21:40.2643233Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161310.xml (deflated 44%) 2022-09-27T16:21:40.2643826Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161313.xml (deflated 44%) 2022-09-27T16:21:40.2644411Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161316.xml (deflated 42%) 2022-09-27T16:21:40.2645098Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161318.xml (deflated 42%) 2022-09-27T16:21:40.2645695Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161319.xml (deflated 42%) 2022-09-27T16:21:40.2646341Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161321.xml (deflated 45%) 2022-09-27T16:21:40.2646926Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161322.xml (deflated 45%) 2022-09-27T16:21:40.2647524Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161324.xml (deflated 41%) 2022-09-27T16:21:40.2648119Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161325.xml (deflated 42%) 2022-09-27T16:21:40.2648719Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161327.xml (deflated 42%) 2022-09-27T16:21:40.2649305Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161328.xml (deflated 42%) 2022-09-27T16:21:40.2649905Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161329.xml (deflated 42%) 2022-09-27T16:21:40.2650494Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161331.xml (deflated 43%) 2022-09-27T16:21:40.2651094Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161332.xml (deflated 42%) 2022-09-27T16:21:40.2651686Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161334.xml (deflated 41%) 2022-09-27T16:21:40.2652271Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161335.xml (deflated 41%) 2022-09-27T16:21:40.2652862Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161337.xml (deflated 41%) 2022-09-27T16:21:40.2653463Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161338.xml (deflated 41%) 2022-09-27T16:21:40.2654055Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161339.xml (deflated 41%) 2022-09-27T16:21:40.2654640Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161341.xml (deflated 43%) 2022-09-27T16:21:40.2655234Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161342.xml (deflated 42%) 2022-09-27T16:21:40.2655832Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161344.xml (deflated 42%) 2022-09-27T16:21:40.2656429Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161345.xml (deflated 42%) 2022-09-27T16:21:40.2657017Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161347.xml (deflated 42%) 2022-09-27T16:21:40.2657611Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161348.xml (deflated 41%) 2022-09-27T16:21:40.2658205Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161349.xml (deflated 41%) 2022-09-27T16:21:40.2658799Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161353.xml (deflated 43%) 2022-09-27T16:21:40.2659378Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161354.xml (deflated 42%) 2022-09-27T16:21:40.2660000Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161356.xml (deflated 42%) 2022-09-27T16:21:40.2660626Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161357.xml (deflated 43%) 2022-09-27T16:21:40.2661222Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161358.xml (deflated 42%) 2022-09-27T16:21:40.2661807Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161400.xml (deflated 42%) 2022-09-27T16:21:40.2662398Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161401.xml (deflated 42%) 2022-09-27T16:21:40.2662990Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161403.xml (deflated 43%) 2022-09-27T16:21:40.2663592Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161404.xml (deflated 43%) 2022-09-27T16:21:40.2664174Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161406.xml (deflated 43%) 2022-09-27T16:21:40.2664767Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161407.xml (deflated 42%) 2022-09-27T16:21:40.2665363Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161409.xml (deflated 44%) 2022-09-27T16:21:40.2665957Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161410.xml (deflated 42%) 2022-09-27T16:21:40.2666540Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161411.xml (deflated 43%) 2022-09-27T16:21:40.2667138Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161413.xml (deflated 41%) 2022-09-27T16:21:40.2667731Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161414.xml (deflated 42%) 2022-09-27T16:21:40.2668330Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161416.xml (deflated 42%) 2022-09-27T16:21:40.2668908Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161419.xml (deflated 42%) 2022-09-27T16:21:40.2669491Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161422.xml (deflated 41%) 2022-09-27T16:21:40.2670059Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161424.xml (deflated 41%) 2022-09-27T16:21:40.2670702Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161425.xml (deflated 41%) 2022-09-27T16:21:40.2671259Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161427.xml (deflated 41%) 2022-09-27T16:21:40.2671831Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161428.xml (deflated 41%) 2022-09-27T16:21:40.2672400Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161430.xml (deflated 41%) 2022-09-27T16:21:40.2672973Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161431.xml (deflated 41%) 2022-09-27T16:21:40.2673526Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161432.xml (deflated 42%) 2022-09-27T16:21:40.2674092Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161434.xml (deflated 42%) 2022-09-27T16:21:40.2674853Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220927161435.xml (deflated 41%) 2022-09-27T16:21:40.2675511Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161437.xml (deflated 41%) 2022-09-27T16:21:40.2676070Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161438.xml (deflated 41%) 2022-09-27T16:21:40.2676640Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161440.xml (deflated 41%) 2022-09-27T16:21:40.2677213Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161441.xml (deflated 41%) 2022-09-27T16:21:40.2677780Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161442.xml (deflated 41%) 2022-09-27T16:21:40.2678329Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161444.xml (deflated 41%) 2022-09-27T16:21:40.2678896Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161445.xml (deflated 41%) 2022-09-27T16:21:40.2679460Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161447.xml (deflated 41%) 2022-09-27T16:21:40.2680019Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161448.xml (deflated 41%) 2022-09-27T16:21:40.2680567Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161450.xml (deflated 41%) 2022-09-27T16:21:40.2681126Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161451.xml (deflated 41%) 2022-09-27T16:21:40.2681693Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161452.xml (deflated 41%) 2022-09-27T16:21:40.2682253Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161454.xml (deflated 41%) 2022-09-27T16:21:40.2682805Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161455.xml (deflated 41%) 2022-09-27T16:21:40.2683365Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161457.xml (deflated 41%) 2022-09-27T16:21:40.2683924Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161458.xml (deflated 41%) 2022-09-27T16:21:40.2684488Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161500.xml (deflated 41%) 2022-09-27T16:21:40.2685034Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161501.xml (deflated 42%) 2022-09-27T16:21:40.2685592Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220927161502.xml (deflated 41%) 2022-09-27T16:21:40.2686125Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220927161504.xml (deflated 40%) 2022-09-27T16:21:40.2686649Z adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20220927161506.xml (deflated 87%) 2022-09-27T16:21:40.2687168Z adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20220927161506.xml (deflated 91%) 2022-09-27T16:21:40.2687735Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220927161547.xml (deflated 86%) 2022-09-27T16:21:40.2688332Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220927161547.xml (deflated 87%) 2022-09-27T16:21:40.2688935Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20220927161616.xml (deflated 81%) 2022-09-27T16:21:40.2689531Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220927161643.xml (deflated 55%) 2022-09-27T16:21:40.2690233Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220927161643.xml (deflated 80%) 2022-09-27T16:21:40.2690897Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220927161643.xml (deflated 79%) 2022-09-27T16:21:40.2691510Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220927161643.xml (deflated 63%) 2022-09-27T16:21:40.2692057Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220927161705.xml (deflated 40%) 2022-09-27T16:21:40.2692564Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220927161707.xml (deflated 40%) 2022-09-27T16:21:40.2693080Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220927161708.xml (deflated 40%) 2022-09-27T16:21:40.2693596Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220927161710.xml (deflated 40%) 2022-09-27T16:21:40.2694119Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220927161712.xml (deflated 40%) 2022-09-27T16:21:40.2694670Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220927161713.xml (deflated 40%) 2022-09-27T16:21:40.2695211Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20220927161715.xml (deflated 40%) 2022-09-27T16:21:40.2695753Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220927161717.xml (deflated 40%) 2022-09-27T16:21:40.2696291Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220927161718.xml (deflated 40%) 2022-09-27T16:21:40.2696827Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220927161720.xml (deflated 40%) 2022-09-27T16:21:40.2697363Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220927161722.xml (deflated 40%) 2022-09-27T16:21:40.2697903Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220927161723.xml (deflated 40%) 2022-09-27T16:21:40.2698423Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220927161725.xml (deflated 39%) 2022-09-27T16:21:40.2698958Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927161727.xml (deflated 40%) 2022-09-27T16:21:40.2699500Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927161728.xml (deflated 39%) 2022-09-27T16:21:40.2700033Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927161730.xml (deflated 39%) 2022-09-27T16:21:40.2700552Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927161732.xml (deflated 40%) 2022-09-27T16:21:40.2701077Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220927161744.xml (deflated 39%) 2022-09-27T16:21:40.2701602Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220927161746.xml (deflated 39%) 2022-09-27T16:21:40.2702119Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161747.xml (deflated 39%) 2022-09-27T16:21:40.2702621Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161749.xml (deflated 39%) 2022-09-27T16:21:40.2703139Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161751.xml (deflated 39%) 2022-09-27T16:21:40.2703686Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161752.xml (deflated 38%) 2022-09-27T16:21:40.2704239Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161754.xml (deflated 38%) 2022-09-27T16:21:40.2704737Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161756.xml (deflated 39%) 2022-09-27T16:21:40.2705250Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161758.xml (deflated 39%) 2022-09-27T16:21:40.2705759Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927161801.xml (deflated 39%) 2022-09-27T16:21:40.2706285Z adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220927161803.xml (deflated 79%) 2022-09-27T16:21:40.2706846Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20220927161819.xml (deflated 84%) 2022-09-27T16:21:40.2707460Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20220927161834.xml (deflated 86%) 2022-09-27T16:21:40.2708020Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220927161851.xml (deflated 38%) 2022-09-27T16:21:40.2708594Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927161854.xml (deflated 42%) 2022-09-27T16:21:40.2709198Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927161856.xml (deflated 40%) 2022-09-27T16:21:40.2709817Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927161858.xml (deflated 41%) 2022-09-27T16:21:40.2710430Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927161900.xml (deflated 42%) 2022-09-27T16:21:40.2711133Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927161901.xml (deflated 41%) 2022-09-27T16:21:40.2711776Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927161905.xml (deflated 41%) 2022-09-27T16:21:40.2712424Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927161909.xml (deflated 41%) 2022-09-27T16:21:40.2713067Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927161913.xml (deflated 41%) 2022-09-27T16:21:40.2713703Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220927161916.xml (deflated 54%) 2022-09-27T16:21:40.2714383Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20220927161928.xml (deflated 68%) 2022-09-27T16:21:40.2715256Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20220927161928.xml (deflated 42%) 2022-09-27T16:21:40.2716074Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20220927161928.xml (deflated 45%) 2022-09-27T16:21:40.2716815Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap/TEST-TestFSDPExecOrder-20220927161938.xml (deflated 77%) 2022-09-27T16:21:40.2717471Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20220927161946.xml (deflated 44%) 2022-09-27T16:21:40.2718134Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20220927161946.xml (deflated 43%) 2022-09-27T16:21:40.2718862Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20220927161946.xml (deflated 61%) 2022-09-27T16:21:40.2719522Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20220927161946.xml (deflated 60%) 2022-09-27T16:21:40.2720143Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20220927161946.xml (deflated 59%) 2022-09-27T16:21:40.2720748Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20220927161946.xml (deflated 62%) 2022-09-27T16:21:40.2721362Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20220927161946.xml (deflated 62%) 2022-09-27T16:21:40.2722004Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20220927161946.xml (deflated 90%) 2022-09-27T16:21:40.2722673Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20220927161946.xml (deflated 70%) 2022-09-27T16:21:40.2723359Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20220927161946.xml (deflated 88%) 2022-09-27T16:21:40.2724056Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20220927161946.xml (deflated 86%) 2022-09-27T16:21:40.2724777Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20220927161946.xml (deflated 63%) 2022-09-27T16:21:40.2725449Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20220927161952.xml (deflated 71%) 2022-09-27T16:21:40.2726070Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20220927161952.xml (deflated 69%) 2022-09-27T16:21:40.2726721Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20220927161952.xml (deflated 66%) 2022-09-27T16:21:40.2727412Z adding: test/test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20220927161957.xml (deflated 61%) 2022-09-27T16:21:40.2728059Z adding: test/test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220927162002.xml (deflated 71%) 2022-09-27T16:21:40.2728659Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20220927162007.xml (deflated 56%) 2022-09-27T16:21:40.2729215Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20220927162011.xml (deflated 45%) 2022-09-27T16:21:40.2729807Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20220927162014.xml (deflated 41%) 2022-09-27T16:21:40.2730439Z adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220927162018.xml (deflated 67%) 2022-09-27T16:21:40.2731072Z adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220927162018.xml (deflated 78%) 2022-09-27T16:21:40.2731700Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220927162021.xml (deflated 44%) 2022-09-27T16:21:40.2732365Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220927162021.xml (deflated 43%) 2022-09-27T16:21:40.2733024Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_planner/TEST-TestSavePlan-20220927162025.xml (deflated 72%) 2022-09-27T16:21:40.2733614Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_utils/TEST-TestMedatadaIndex-20220927162027.xml (deflated 72%) 2022-09-27T16:21:40.2734232Z adding: test/test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20220927162029.xml (deflated 43%) 2022-09-27T16:21:40.2734889Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220927162030.xml (deflated 76%) 2022-09-27T16:21:40.2735574Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20220927162032.xml (deflated 71%) 2022-09-27T16:21:40.2736210Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220927162035.xml (deflated 42%) 2022-09-27T16:21:40.2736849Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220927162037.xml (deflated 42%) 2022-09-27T16:21:40.2737470Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220927162038.xml (deflated 42%) 2022-09-27T16:21:40.2738098Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220927162040.xml (deflated 42%) 2022-09-27T16:21:40.2738737Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162042.xml (deflated 43%) 2022-09-27T16:21:40.2739384Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162043.xml (deflated 43%) 2022-09-27T16:21:40.2740018Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162045.xml (deflated 43%) 2022-09-27T16:21:40.2740668Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162047.xml (deflated 43%) 2022-09-27T16:21:40.2741311Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162049.xml (deflated 43%) 2022-09-27T16:21:40.2741951Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162050.xml (deflated 43%) 2022-09-27T16:21:40.2742580Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162052.xml (deflated 43%) 2022-09-27T16:21:40.2743217Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162054.xml (deflated 43%) 2022-09-27T16:21:40.2743856Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220927162055.xml (deflated 42%) 2022-09-27T16:21:40.2744499Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20220927162057.xml (deflated 62%) 2022-09-27T16:21:40.2745114Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20220927162059.xml (deflated 63%) 2022-09-27T16:21:40.2745758Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220927162100.xml (deflated 62%) 2022-09-27T16:21:40.2746399Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20220927162101.xml (deflated 61%) 2022-09-27T16:21:40.2746870Z adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 79%) 2022-09-27T16:21:40.2764838Z ##[group]Run # Remove any previous test reports if they exist 2022-09-27T16:21:40.2765137Z # Remove any previous test reports if they exist 2022-09-27T16:21:40.2765424Z rm -f usage-log-*.zip 2022-09-27T16:21:40.2765690Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2022-09-27T16:21:40.2766035Z # so check to see if the file exists first 2022-09-27T16:21:40.2766266Z if [ -f 'usage_log.txt' ]; then 2022-09-27T16:21:40.2766510Z  zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt' 2022-09-27T16:21:40.2766712Z fi 2022-09-27T16:21:40.2777672Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T16:21:40.2777892Z env: 2022-09-27T16:21:40.2778057Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:40.2778315Z FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_8576161248 2022-09-27T16:21:40.2778552Z ##[endgroup] 2022-09-27T16:21:40.3157626Z adding: usage_log.txt (deflated 94%) 2022-09-27T16:21:40.3215549Z ##[group]Run seemethere/upload-artifact-s3@v5 2022-09-27T16:21:40.3215749Z with: 2022-09-27T16:21:40.3215957Z s3-prefix: pytorch/pytorch/3133193930/2/artifact 2022-09-27T16:21:40.3216167Z retention-days: 14 2022-09-27T16:21:40.3216345Z if-no-files-found: warn 2022-09-27T16:21:40.3216547Z path: test-jsons-*.zip 2022-09-27T16:21:40.3216724Z name: artifact 2022-09-27T16:21:40.3216891Z s3-bucket: gha-artifacts 2022-09-27T16:21:40.3217073Z region: us-east-1 2022-09-27T16:21:40.3217234Z env: 2022-09-27T16:21:40.3217388Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:40.3217568Z ##[endgroup] 2022-09-27T16:21:40.6397675Z NOTE: s3-prefix specified, ignoring name parameter 2022-09-27T16:21:40.6398205Z With the provided path, there will be 1 file uploaded 2022-09-27T16:21:40.6398548Z Uploading to s3 prefix: pytorch/pytorch/3133193930/2/artifact 2022-09-27T16:21:40.6404647Z Starting upload of test-jsons-test-distributed-1-2-linux.2xlarge_8576161248.zip 2022-09-27T16:21:40.9824566Z Finished upload of test-jsons-test-distributed-1-2-linux.2xlarge_8576161248.zip 2022-09-27T16:21:40.9932204Z ##[group]Run seemethere/upload-artifact-s3@v5 2022-09-27T16:21:40.9932430Z with: 2022-09-27T16:21:40.9932634Z s3-prefix: pytorch/pytorch/3133193930/2/artifact 2022-09-27T16:21:40.9932848Z retention-days: 14 2022-09-27T16:21:40.9933044Z if-no-files-found: error 2022-09-27T16:21:40.9933249Z path: test-reports-*.zip 2022-09-27T16:21:40.9933423Z name: artifact 2022-09-27T16:21:40.9933604Z s3-bucket: gha-artifacts 2022-09-27T16:21:40.9933790Z region: us-east-1 2022-09-27T16:21:40.9933954Z env: 2022-09-27T16:21:40.9934112Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:40.9934294Z ##[endgroup] 2022-09-27T16:21:41.3020686Z NOTE: s3-prefix specified, ignoring name parameter 2022-09-27T16:21:41.3020980Z With the provided path, there will be 1 file uploaded 2022-09-27T16:21:41.3021239Z Uploading to s3 prefix: pytorch/pytorch/3133193930/2/artifact 2022-09-27T16:21:41.3027333Z Starting upload of test-reports-test-distributed-1-2-linux.2xlarge_8576161248.zip 2022-09-27T16:21:41.4002752Z Finished upload of test-reports-test-distributed-1-2-linux.2xlarge_8576161248.zip 2022-09-27T16:21:41.4109054Z ##[group]Run seemethere/upload-artifact-s3@v5 2022-09-27T16:21:41.4109273Z with: 2022-09-27T16:21:41.4109486Z s3-prefix: pytorch/pytorch/3133193930/2/artifact 2022-09-27T16:21:41.4109688Z retention-days: 14 2022-09-27T16:21:41.4109881Z if-no-files-found: ignore 2022-09-27T16:21:41.4110078Z path: usage-log-*.zip 2022-09-27T16:21:41.4110244Z name: artifact 2022-09-27T16:21:41.4110426Z s3-bucket: gha-artifacts 2022-09-27T16:21:41.4110696Z region: us-east-1 2022-09-27T16:21:41.4110863Z env: 2022-09-27T16:21:41.4111021Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:41.4111207Z ##[endgroup] 2022-09-27T16:21:41.7204391Z NOTE: s3-prefix specified, ignoring name parameter 2022-09-27T16:21:41.7204912Z With the provided path, there will be 1 file uploaded 2022-09-27T16:21:41.7205413Z Uploading to s3 prefix: pytorch/pytorch/3133193930/2/artifact 2022-09-27T16:21:41.7211322Z Starting upload of usage-log-test-distributed-1-2-linux.2xlarge_8576161248.zip 2022-09-27T16:21:41.8417954Z Finished upload of usage-log-test-distributed-1-2-linux.2xlarge_8576161248.zip 2022-09-27T16:21:41.8529889Z ##[group]Run set -x 2022-09-27T16:21:41.8530174Z set -x 2022-09-27T16:21:41.8530401Z python3 -m pip install -r requirements.txt 2022-09-27T16:21:41.8530653Z python3 -m pip install boto3==1.19.12 2022-09-27T16:21:41.8530949Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-09-27T16:21:41.8542257Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T16:21:41.8542475Z env: 2022-09-27T16:21:41.8542654Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:41.8542845Z AWS_DEFAULT_REGION: us-east-1 2022-09-27T16:21:41.8543038Z BRANCH: pull/85462 2022-09-27T16:21:41.8543222Z TEST_CONFIG: distributed 2022-09-27T16:21:41.8543389Z SHARD_NUMBER: 1 2022-09-27T16:21:41.8543603Z BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7 2022-09-27T16:21:41.8543813Z PR_NUMBER: 85462 2022-09-27T16:21:41.8543989Z PYTORCH_RETRY_TEST_CASES: 1 2022-09-27T16:21:41.8544195Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-09-27T16:21:41.8544447Z SHA1: 52424e2bf38e454d535881fed9628d3e20f4f944 2022-09-27T16:21:41.8544635Z TAG: 2022-09-27T16:21:41.8544797Z WORKFLOW_ID: 3133193930 2022-09-27T16:21:41.8545116Z GITHUB_TOKEN: *** 2022-09-27T16:21:41.8545291Z GHA_WORKFLOW_JOB_ID: 8576161248 2022-09-27T16:21:41.8545480Z ##[endgroup] 2022-09-27T16:21:41.8570120Z + python3 -m pip install -r requirements.txt 2022-09-27T16:21:42.0668863Z Defaulting to user installation because normal site-packages is not writeable 2022-09-27T16:21:42.0925124Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3) 2022-09-27T16:21:42.0949980Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.3) 2022-09-27T16:21:42.0958021Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2) 2022-09-27T16:21:42.0967305Z Requirement already satisfied: hypothesis in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (6.54.6) 2022-09-27T16:21:42.1327645Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.21.6) 2022-09-27T16:21:42.1336033Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1) 2022-09-27T16:21:42.1412475Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.0) 2022-09-27T16:21:42.1419845Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0) 2022-09-27T16:21:42.1635311Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3) 2022-09-27T16:21:42.1798883Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (1.16.0) 2022-09-27T16:21:42.1806911Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (0.6.6) 2022-09-27T16:21:42.1812194Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (4.3.0) 2022-09-27T16:21:42.1821123Z Requirement already satisfied: sympy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.10.1) 2022-09-27T16:21:42.1841511Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.37.1) 2022-09-27T16:21:42.1864417Z Requirement already satisfied: attrs>=19.2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (22.1.0) 2022-09-27T16:21:42.2109915Z Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (2.4.0) 2022-09-27T16:21:42.2119751Z Requirement already satisfied: exceptiongroup>=1.0.0rc8; python_version < "3.11" in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (1.0.0rc9) 2022-09-27T16:21:42.2135176Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24) 2022-09-27T16:21:42.2143134Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.12) 2022-09-27T16:21:42.2296815Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12) 2022-09-27T16:21:42.2316342Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4) 2022-09-27T16:21:42.2326951Z Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/.local/lib/python3.7/site-packages (from sympy->-r requirements.txt (line 14)) (1.2.1) 2022-09-27T16:21:42.2964920Z + python3 -m pip install boto3==1.19.12 2022-09-27T16:21:42.5039000Z Defaulting to user installation because normal site-packages is not writeable 2022-09-27T16:21:42.5217308Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12) 2022-09-27T16:21:42.5265861Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0) 2022-09-27T16:21:42.5278343Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12) 2022-09-27T16:21:42.5319142Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2) 2022-09-27T16:21:42.5352443Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.12) 2022-09-27T16:21:42.5505912Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2) 2022-09-27T16:21:42.5523681Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0) 2022-09-27T16:21:42.7136081Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-09-27T16:21:50.4671044Z [scribe] Scribe access token not provided, sending report via boto3... 2022-09-27T16:21:50.4671435Z 2022-09-27T16:21:50.4671734Z ----- Historic stats comparison result ------ 2022-09-27T16:21:50.4671910Z 2022-09-27T16:21:50.4672060Z job: linux-focal-py3.7-gcc7 2022-09-27T16:21:50.4672296Z commit: 52424e2bf38e454d535881fed9628d3e20f4f944 2022-09-27T16:21:50.4672438Z 2022-09-27T16:21:50.4672586Z Commit graph (base is most recent master ancestor with at least one S3 report): 2022-09-27T16:21:50.4672751Z 2022-09-27T16:21:50.4672820Z : (master) 2022-09-27T16:21:50.4672985Z | 2022-09-27T16:21:50.4673264Z | * 52424e2bf3 (HEAD) total time 1836.85s 2022-09-27T16:21:50.4673588Z | | 2022-09-27T16:21:50.4673870Z | : (4 commits) 2022-09-27T16:21:50.4674183Z |/ 2022-09-27T16:21:50.4677247Z * c7c2578f93 (base) 6 reports, total time 2509.28s ± 1765.63s 2022-09-27T16:21:50.4677730Z * 99ad8a3048 6 reports, total time 2509.77s ± 1771.89s 2022-09-27T16:21:50.4678056Z * 34296e2f4c 6 reports, total time 2525.78s ± 1741.22s 2022-09-27T16:21:50.4678622Z * 4523ac7aa1 6 reports, total time 2546.75s ± 1751.11s 2022-09-27T16:21:50.4679014Z * f21e77d9a6 6 reports, total time 2521.77s ± 1744.10s 2022-09-27T16:21:50.4679332Z * 26a861cb27 6 reports, total time 2538.71s ± 1776.59s 2022-09-27T16:21:50.4679645Z * 56a41b5998 6 reports, total time 2537.11s ± 1781.85s 2022-09-27T16:21:50.4679942Z * 1910c5847e 6 reports, total time 2539.59s ± 1745.17s 2022-09-27T16:21:50.4680258Z * caa0ab557d 6 reports, total time 2502.08s ± 1718.61s 2022-09-27T16:21:50.4680483Z * 0336308be5 0 reports 2022-09-27T16:21:50.4680660Z | 2022-09-27T16:21:50.4680797Z : 2022-09-27T16:21:50.4680892Z 2022-09-27T16:21:50.4681013Z Removed (across 799 suites) 0 tests, totaling 0.00s 2022-09-27T16:21:50.4681278Z Modified (across 0 suites) 0 tests, totaling 0.00s 2022-09-27T16:21:50.4681527Z Added (across 96 suites) 1122 tests, totaling +1836.85s 2022-09-27T16:21:50.5215138Z Prepare all required actions 2022-09-27T16:21:50.5235162Z ##[group]Run ./.github/actions/teardown-linux 2022-09-27T16:21:50.5235379Z with: 2022-09-27T16:21:50.5235519Z env: 2022-09-27T16:21:50.5235692Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:50.5235877Z ##[endgroup] 2022-09-27T16:21:50.5250133Z ##[group]Run set -eou pipefail 2022-09-27T16:21:50.5250355Z set -eou pipefail 2022-09-27T16:21:50.5250535Z  2022-09-27T16:21:50.5250758Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2022-09-27T16:21:50.5251009Z for _ in $(seq 1440); do 2022-09-27T16:21:50.5251230Z  # Break if no ssh session exists anymore 2022-09-27T16:21:50.5251433Z  if [ "$(who)" = "" ]; then 2022-09-27T16:21:50.5251612Z  break 2022-09-27T16:21:50.5251777Z  fi 2022-09-27T16:21:50.5251945Z  echo "." 2022-09-27T16:21:50.5252105Z  sleep 5 2022-09-27T16:21:50.5252300Z done 2022-09-27T16:21:50.5263359Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T16:21:50.5263580Z env: 2022-09-27T16:21:50.5263754Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:50.5263925Z ##[endgroup] 2022-09-27T16:21:50.5287506Z Holding runner for 2 hours until all ssh sessions have logged out 2022-09-27T16:21:50.5325677Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2022-09-27T16:21:50.5325987Z # ignore expansion of "docker ps -q" since it could be empty 2022-09-27T16:21:50.5326239Z # shellcheck disable=SC2046 2022-09-27T16:21:50.5326451Z docker stop $(docker ps -q) || true 2022-09-27T16:21:50.5326681Z # Prune all of the docker images 2022-09-27T16:21:50.5326899Z docker system prune -af 2022-09-27T16:21:50.5337684Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-09-27T16:21:50.5337902Z env: 2022-09-27T16:21:50.5338081Z GIT_DEFAULT_BRANCH: master 2022-09-27T16:21:50.5338268Z ##[endgroup] 2022-09-27T16:21:50.9478906Z 51c56aedd00a 2022-09-27T16:21:51.4798267Z Deleted Containers: 2022-09-27T16:21:51.4798813Z 51c56aedd00ae5cc22b203da5b2416a90eefb59e9fc34e33dc8868914cc83ee3 2022-09-27T16:21:51.4798999Z 2022-09-27T16:21:54.0389999Z Deleted Images: 2022-09-27T16:21:54.0391064Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29 2022-09-27T16:21:54.0391753Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7@sha256:aa31a48de2d62af84cacd725abaa1599e63c6bb3385fa6e176b6d91f90a39348 2022-09-27T16:21:54.0392260Z deleted: sha256:09cc5e0dbbe1120475828f67580c1e4ef0ac166558f7a9e82119c9daf09ed17b 2022-09-27T16:21:54.0392602Z deleted: sha256:4f4d0a1da9af50d9a8bd02155aaea8b254a8039b940db877e78a5b25776e1f97 2022-09-27T16:21:54.0392907Z deleted: sha256:40273608be9f35547c534c65183771d8d024a327b2f8231776e449a0c4be59e4 2022-09-27T16:21:54.0393223Z deleted: sha256:295b500c49641b27efb539c901bd567f2b8738ab1f67465c28c5209fef872936 2022-09-27T16:21:54.0393540Z deleted: sha256:aff7d79df966d7c1c7cf29b78408592957f0317054355d087aa59098aa32abda 2022-09-27T16:21:54.0394024Z deleted: sha256:0e0419c0016bdfbf589e8507dad2150575b0a84dd2b27be7703cfa05e2139c5b 2022-09-27T16:21:54.0394348Z deleted: sha256:0e1f7b9f93cf30ea54c8e6f1485a1a3fd9776d17cc90f1bfb5ce6b5b9185296e 2022-09-27T16:21:54.0394832Z deleted: sha256:d3deacfa23bfb0920ec3a625c1ee1963513159aaa5463910dd41bafa8e54de71 2022-09-27T16:21:54.0395171Z deleted: sha256:e66d036a902f9f1d15a19abacd7321bba3d8869b725e99b43a5d20fd6fdb01c9 2022-09-27T16:21:54.0395487Z deleted: sha256:c5e59df58f1aab354a2874070ad89a043c3178016eab17887ed9237f91887294 2022-09-27T16:21:54.0395807Z deleted: sha256:0d8f363a24f0b380db49f8fa69e9da0537d3a51ae051248dfd3563de574ca15d 2022-09-27T16:21:54.0396137Z deleted: sha256:55bb4ae9fce9854a9f8345d077cd37423d7b6f050034cadadbbf1974ff155061 2022-09-27T16:21:54.0396515Z deleted: sha256:124538c5717f1c1615e490bfd34cab23dc00516fbd43b4dbd0755e1a5119474b 2022-09-27T16:21:54.0396836Z deleted: sha256:98e71d41b4fb04ffa587728b6f070c0d922a9909fb272edae447c0ed732b17c6 2022-09-27T16:21:54.0397153Z deleted: sha256:dd100ce53375a22379598335be3a232a3eb20465df4376afdd481c3ab0a01666 2022-09-27T16:21:54.0397470Z deleted: sha256:06da69152df74c7a52462b8f1f7adf2d50e5f84ae02c20a48597f844e24832f4 2022-09-27T16:21:54.0397776Z deleted: sha256:7f90b14cc4e879de499e9c152c229852c1a71e1c6bb0216e21b433d34e401be0 2022-09-27T16:21:54.0398087Z deleted: sha256:2a332f12721f3199efd9441306d5d38a6f031cdfdf68beb90afb40116f7ac1ad 2022-09-27T16:21:54.0398411Z deleted: sha256:9c866de535edd7886f74437d3fbcafd6d58fe3322aa4af8606e46e720f022e9c 2022-09-27T16:21:54.0398726Z deleted: sha256:097a48bd86a6a3542f566d38833d93ded3bc082bdad7e4026debfbe895bb5105 2022-09-27T16:21:54.0399043Z deleted: sha256:a8b120c70115f75a61a81c96d1abef9e741618d32124b4994375a805a5941958 2022-09-27T16:21:54.0399352Z deleted: sha256:15a928b5307c0332ba50caffe867123de36c74475d7861b8cdf2dbfd58dd5320 2022-09-27T16:21:54.0399692Z deleted: sha256:eb93d3b8365b2da69ddeb34c24e5ba2fae6eb2fad4b6fadae3e298a9a0b26af2 2022-09-27T16:21:54.0400027Z deleted: sha256:e2ab13aa2f895ecf403ea9781db734f7b21a20aeb1478e32c5416f969604dc89 2022-09-27T16:21:54.0400337Z deleted: sha256:6faa89577c8237b30a828e6382086b71579a76333d2d79839246ed4c42daf433 2022-09-27T16:21:54.0400642Z deleted: sha256:94f3fc0358a54e1bab1cff99f032187d852d5a9e21539debb385d3e965e0f811 2022-09-27T16:21:54.0400950Z deleted: sha256:0890fc3c642cc4c11d7b3b400a3f6af80996973f0d1524071eb15cbdd5431266 2022-09-27T16:21:54.0401271Z deleted: sha256:ec7d767c32abaf566f084ffeb2d37bda985f13102952e21948f7b055ccccf7ab 2022-09-27T16:21:54.0401579Z deleted: sha256:2758607d82553d625e21563dec5e252097fd6804de79dd60b78d9452d8f0e384 2022-09-27T16:21:54.0401878Z deleted: sha256:1c30230d1bc09ce084c2c0d3c24c81e39a270b3359be353f188f13809b2e95ff 2022-09-27T16:21:54.0402172Z deleted: sha256:f66b31a4878493b2a4cf4b457a461d78c1275ce0b853312e14b5b205f5f54b12 2022-09-27T16:21:54.0402490Z deleted: sha256:c96b1aa08b8cdf00ede17699be953c554e540b97216fba81a59a101860c877c6 2022-09-27T16:21:54.0402811Z deleted: sha256:0853e57dc641a08006c8b8cd96caa8755d3981706b49748776bf6c7ee23abe6c 2022-09-27T16:21:54.0403125Z deleted: sha256:a88ebc418ab3040ebaafd029a4c80439225b1ce939f383ec70c503fff55528ee 2022-09-27T16:21:54.0403443Z deleted: sha256:5a86190b95485dba13fec16a89446426425223587ad649a56b4faf0f64305120 2022-09-27T16:21:54.0403753Z deleted: sha256:016eff1a025a65973028af96cf1fbe6c4b0e1c258e04e6a2dec976062924c0cf 2022-09-27T16:21:54.0404083Z deleted: sha256:3ce5ce7ae3df63641c9b8859e93e8f870c8bcfadc998c17b2169676be5e0812d 2022-09-27T16:21:54.0404396Z deleted: sha256:b161ee8936514898f92131df4d7de87314f3ed0639e31bbd2c3e83bdea170f16 2022-09-27T16:21:54.0404711Z deleted: sha256:e6a3a6ca01553128af3686d5bface6135e56f897e0f9ba754e9b924d446105e6 2022-09-27T16:21:54.0405038Z deleted: sha256:0cd4c1b082ed42bb2648c766b0746e3eb8771eafc8ce4130c649f3d29b634162 2022-09-27T16:21:54.0405338Z deleted: sha256:76a601a88697897790d443b0214d908f489395a5d4ea98726caf0a80a66ecd19 2022-09-27T16:21:54.0405643Z deleted: sha256:e9a0ce4d7b4ea637152093c92c0cbe039a7e810c4aa90c848f312c10262b974d 2022-09-27T16:21:54.0406027Z deleted: sha256:e4e7cd35b02c90bb1da4c482c1f10f82c7ef6daae84f9ceeac9815d19ce51dbd 2022-09-27T16:21:54.0406365Z deleted: sha256:8e15ae424f8839bb14c5629f1fec3c93cebec42642eabeabd208ca359a19db66 2022-09-27T16:21:54.0406687Z deleted: sha256:e43a9a37dcae4c22f906f9f4450ad78b70905cd289f7ad9f9d40d7c8b12d8f72 2022-09-27T16:21:54.0407000Z deleted: sha256:06bcf3094318f05722b94bc725806fc0d2367a7865af5b199f0567a72699f757 2022-09-27T16:21:54.0407312Z deleted: sha256:51f25a037328b3be1af44473dfa7a876a5355f74cef818b46bf003c33b1ec2c5 2022-09-27T16:21:54.0407621Z deleted: sha256:b17dd3db6bb4f4513025c933f6a63f1ed0921065fe34fd195c40c4acff4f696c 2022-09-27T16:21:54.0407942Z deleted: sha256:71ca3ec58848585b585c80e00986d5e8789e8169554706949b8a3d8dcdf72e3b 2022-09-27T16:21:54.0408248Z deleted: sha256:6f4b9ef5ee757f0395c054927ff65df65438bdf23c05109f7f3107fb94719eef 2022-09-27T16:21:54.0408617Z deleted: sha256:883eb0bcef572233ec6334ad982f16d8ba42bcffee0735ae8969fe257019481d 2022-09-27T16:21:54.0408932Z deleted: sha256:f0d75587c52d51e623884d33f223de82faa346a03a9e34271eaf6765e8995d2a 2022-09-27T16:21:54.0409239Z deleted: sha256:25ab06e14d5870622fe0c785fa28eedb95787ed7286d8c665289672a2e11334a 2022-09-27T16:21:54.0409565Z deleted: sha256:f9bb5b08ac5635a4a64adbfe88fe48bcb0ed9425ab68ecf412265012933639e6 2022-09-27T16:21:54.0409901Z deleted: sha256:aa9ab2200af3ee6cbe32d9703272a4a85de2def9b0c9a20bd09f3dca10f0e47d 2022-09-27T16:21:54.0410236Z deleted: sha256:30936d838a38cd0a0c2ddc832a2a17af9d84327e0f23cac0e773686bfed7e6e1 2022-09-27T16:21:54.0410558Z deleted: sha256:5c5425a2d9bb189d0f99b1c37511d370d599afca6b8b0d3348b3b2ace341ddaf 2022-09-27T16:21:54.0410885Z deleted: sha256:998fb9c2fa230ea5cc1f997926614d854d193df0b2129b2b12d7ede4e5d43f6e 2022-09-27T16:21:54.0411188Z deleted: sha256:b40ed86654e59e1012e1716d5384910f8c3bb58274b7b00fca564a53e9897ba3 2022-09-27T16:21:54.0411356Z 2022-09-27T16:21:54.0414634Z Total reclaimed space: 7.171GB 2022-09-27T16:21:54.0467696Z Post job cleanup. 2022-09-27T16:21:54.0495843Z Post job cleanup. 2022-09-27T16:21:54.1469254Z [command]/usr/bin/git version 2022-09-27T16:21:54.1507475Z git version 2.37.1 2022-09-27T16:21:54.1546236Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/6275602c-9bdf-421c-96dd-41c94cae4932' before making global git config changes 2022-09-27T16:21:54.1546664Z Adding repository directory to the temporary git global config as a safe directory 2022-09-27T16:21:54.1552429Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-09-27T16:21:54.1590785Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-09-27T16:21:54.1621975Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-09-27T16:21:54.1880638Z Entering 'android/libs/fbjni' 2022-09-27T16:21:54.1914177Z Entering 'third_party/FP16' 2022-09-27T16:21:54.1949467Z Entering 'third_party/FXdiv' 2022-09-27T16:21:54.1981896Z Entering 'third_party/NNPACK' 2022-09-27T16:21:54.2017586Z Entering 'third_party/QNNPACK' 2022-09-27T16:21:54.2051511Z Entering 'third_party/VulkanMemoryAllocator' 2022-09-27T16:21:54.2084538Z Entering 'third_party/XNNPACK' 2022-09-27T16:21:54.2129951Z Entering 'third_party/benchmark' 2022-09-27T16:21:54.2164152Z Entering 'third_party/cpuinfo' 2022-09-27T16:21:54.2197902Z Entering 'third_party/cub' 2022-09-27T16:21:54.2232848Z Entering 'third_party/cudnn_frontend' 2022-09-27T16:21:54.2272322Z Entering 'third_party/cutlass' 2022-09-27T16:21:54.2312235Z Entering 'third_party/eigen' 2022-09-27T16:21:54.2347302Z Entering 'third_party/fbgemm' 2022-09-27T16:21:54.2381185Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-09-27T16:21:54.2415357Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-09-27T16:21:54.2448188Z Entering 'third_party/fbgemm/third_party/googletest' 2022-09-27T16:21:54.2480903Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-09-27T16:21:54.2515704Z Entering 'third_party/flatbuffers' 2022-09-27T16:21:54.2551482Z Entering 'third_party/fmt' 2022-09-27T16:21:54.2585501Z Entering 'third_party/foxi' 2022-09-27T16:21:54.2619261Z Entering 'third_party/gemmlowp/gemmlowp' 2022-09-27T16:21:54.2655854Z Entering 'third_party/gloo' 2022-09-27T16:21:54.2689635Z Entering 'third_party/googletest' 2022-09-27T16:21:54.2724222Z Entering 'third_party/ideep' 2022-09-27T16:21:54.2756701Z Entering 'third_party/ideep/mkl-dnn' 2022-09-27T16:21:54.2791107Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-09-27T16:21:54.2830355Z Entering 'third_party/ios-cmake' 2022-09-27T16:21:54.2864509Z Entering 'third_party/ittapi' 2022-09-27T16:21:54.2899081Z Entering 'third_party/kineto' 2022-09-27T16:21:54.2932787Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-09-27T16:21:54.2966644Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-09-27T16:21:54.3001929Z Entering 'third_party/nccl/nccl' 2022-09-27T16:21:54.3036129Z Entering 'third_party/neon2sse' 2022-09-27T16:21:54.3069089Z Entering 'third_party/nlohmann' 2022-09-27T16:21:54.3105146Z Entering 'third_party/onnx' 2022-09-27T16:21:54.3150532Z Entering 'third_party/onnx/third_party/benchmark' 2022-09-27T16:21:54.3184840Z Entering 'third_party/onnx/third_party/pybind11' 2022-09-27T16:21:54.3221586Z Entering 'third_party/onnx-tensorrt' 2022-09-27T16:21:54.3255739Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-09-27T16:21:54.3294569Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-09-27T16:21:54.3330417Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-09-27T16:21:54.3365651Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-09-27T16:21:54.3406841Z Entering 'third_party/pocketfft' 2022-09-27T16:21:54.3441081Z Entering 'third_party/protobuf' 2022-09-27T16:21:54.3478471Z Entering 'third_party/protobuf/third_party/benchmark' 2022-09-27T16:21:54.3512047Z Entering 'third_party/protobuf/third_party/googletest' 2022-09-27T16:21:54.3548163Z Entering 'third_party/psimd' 2022-09-27T16:21:54.3583573Z Entering 'third_party/pthreadpool' 2022-09-27T16:21:54.3618179Z Entering 'third_party/pybind11' 2022-09-27T16:21:54.3652948Z Entering 'third_party/python-enum' 2022-09-27T16:21:54.3687098Z Entering 'third_party/python-peachpy' 2022-09-27T16:21:54.3720656Z Entering 'third_party/python-six' 2022-09-27T16:21:54.3754651Z Entering 'third_party/sleef' 2022-09-27T16:21:54.3789603Z Entering 'third_party/tbb' 2022-09-27T16:21:54.3826650Z Entering 'third_party/tensorpipe' 2022-09-27T16:21:54.3862610Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-09-27T16:21:54.3897929Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-09-27T16:21:54.3932407Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-09-27T16:21:54.3967377Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-09-27T16:21:54.4003321Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-09-27T16:21:54.4040156Z Entering 'third_party/zstd' 2022-09-27T16:21:54.4090788Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-09-27T16:21:54.4119004Z http.https://github.com/.extraheader 2022-09-27T16:21:54.4127488Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2022-09-27T16:21:54.4162144Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-09-27T16:21:54.4422562Z Entering 'android/libs/fbjni' 2022-09-27T16:21:54.4442012Z http.https://github.com/.extraheader 2022-09-27T16:21:54.4469385Z Entering 'third_party/FP16' 2022-09-27T16:21:54.4490160Z http.https://github.com/.extraheader 2022-09-27T16:21:54.4517729Z Entering 'third_party/FXdiv' 2022-09-27T16:21:54.4537618Z http.https://github.com/.extraheader 2022-09-27T16:21:54.4563608Z Entering 'third_party/NNPACK' 2022-09-27T16:21:54.4583129Z http.https://github.com/.extraheader 2022-09-27T16:21:54.4609737Z Entering 'third_party/QNNPACK' 2022-09-27T16:21:54.4630051Z http.https://github.com/.extraheader 2022-09-27T16:21:54.4657120Z Entering 'third_party/VulkanMemoryAllocator' 2022-09-27T16:21:54.4677014Z http.https://github.com/.extraheader 2022-09-27T16:21:54.4703545Z Entering 'third_party/XNNPACK' 2022-09-27T16:21:54.4724715Z http.https://github.com/.extraheader 2022-09-27T16:21:54.4761524Z Entering 'third_party/benchmark' 2022-09-27T16:21:54.4781291Z http.https://github.com/.extraheader 2022-09-27T16:21:54.4808583Z Entering 'third_party/cpuinfo' 2022-09-27T16:21:54.4828059Z http.https://github.com/.extraheader 2022-09-27T16:21:54.4855914Z Entering 'third_party/cub' 2022-09-27T16:21:54.4876161Z http.https://github.com/.extraheader 2022-09-27T16:21:54.4903127Z Entering 'third_party/cudnn_frontend' 2022-09-27T16:21:54.4924486Z http.https://github.com/.extraheader 2022-09-27T16:21:54.4956268Z Entering 'third_party/cutlass' 2022-09-27T16:21:54.4975806Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5010212Z Entering 'third_party/eigen' 2022-09-27T16:21:54.5028904Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5058316Z Entering 'third_party/fbgemm' 2022-09-27T16:21:54.5079940Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5106833Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-09-27T16:21:54.5126596Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5153087Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-09-27T16:21:54.5172781Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5199667Z Entering 'third_party/fbgemm/third_party/googletest' 2022-09-27T16:21:54.5219390Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5247339Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-09-27T16:21:54.5266622Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5295137Z Entering 'third_party/flatbuffers' 2022-09-27T16:21:54.5315485Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5344098Z Entering 'third_party/fmt' 2022-09-27T16:21:54.5364221Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5391066Z Entering 'third_party/foxi' 2022-09-27T16:21:54.5411319Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5438125Z Entering 'third_party/gemmlowp/gemmlowp' 2022-09-27T16:21:54.5457427Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5484766Z Entering 'third_party/gloo' 2022-09-27T16:21:54.5503557Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5531127Z Entering 'third_party/googletest' 2022-09-27T16:21:54.5551693Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5579434Z Entering 'third_party/ideep' 2022-09-27T16:21:54.5599865Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5626025Z Entering 'third_party/ideep/mkl-dnn' 2022-09-27T16:21:54.5645880Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5674958Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-09-27T16:21:54.5695339Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5728128Z Entering 'third_party/ios-cmake' 2022-09-27T16:21:54.5747440Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5774303Z Entering 'third_party/ittapi' 2022-09-27T16:21:54.5794574Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5821817Z Entering 'third_party/kineto' 2022-09-27T16:21:54.5842178Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5868177Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-09-27T16:21:54.5888667Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5914825Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-09-27T16:21:54.5934238Z http.https://github.com/.extraheader 2022-09-27T16:21:54.5962607Z Entering 'third_party/nccl/nccl' 2022-09-27T16:21:54.5982881Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6009471Z Entering 'third_party/neon2sse' 2022-09-27T16:21:54.6028666Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6055228Z Entering 'third_party/nlohmann' 2022-09-27T16:21:54.6077084Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6102882Z Entering 'third_party/onnx' 2022-09-27T16:21:54.6122445Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6159541Z Entering 'third_party/onnx/third_party/benchmark' 2022-09-27T16:21:54.6179883Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6207234Z Entering 'third_party/onnx/third_party/pybind11' 2022-09-27T16:21:54.6227085Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6255757Z Entering 'third_party/onnx-tensorrt' 2022-09-27T16:21:54.6275389Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6300951Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-09-27T16:21:54.6321496Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6352716Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-09-27T16:21:54.6371628Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6398675Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-09-27T16:21:54.6417228Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6443401Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-09-27T16:21:54.6462443Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6494868Z Entering 'third_party/pocketfft' 2022-09-27T16:21:54.6515386Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6541377Z Entering 'third_party/protobuf' 2022-09-27T16:21:54.6560862Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6590248Z Entering 'third_party/protobuf/third_party/benchmark' 2022-09-27T16:21:54.6609824Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6635925Z Entering 'third_party/protobuf/third_party/googletest' 2022-09-27T16:21:54.6655005Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6684560Z Entering 'third_party/psimd' 2022-09-27T16:21:54.6704942Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6730426Z Entering 'third_party/pthreadpool' 2022-09-27T16:21:54.6750214Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6776449Z Entering 'third_party/pybind11' 2022-09-27T16:21:54.6797824Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6823964Z Entering 'third_party/python-enum' 2022-09-27T16:21:54.6843150Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6868853Z Entering 'third_party/python-peachpy' 2022-09-27T16:21:54.6888898Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6915068Z Entering 'third_party/python-six' 2022-09-27T16:21:54.6935904Z http.https://github.com/.extraheader 2022-09-27T16:21:54.6962011Z Entering 'third_party/sleef' 2022-09-27T16:21:54.6981683Z http.https://github.com/.extraheader 2022-09-27T16:21:54.7008053Z Entering 'third_party/tbb' 2022-09-27T16:21:54.7028078Z http.https://github.com/.extraheader 2022-09-27T16:21:54.7056053Z Entering 'third_party/tensorpipe' 2022-09-27T16:21:54.7076357Z http.https://github.com/.extraheader 2022-09-27T16:21:54.7102599Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-09-27T16:21:54.7122031Z http.https://github.com/.extraheader 2022-09-27T16:21:54.7148984Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-09-27T16:21:54.7168287Z http.https://github.com/.extraheader 2022-09-27T16:21:54.7193767Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-09-27T16:21:54.7212602Z http.https://github.com/.extraheader 2022-09-27T16:21:54.7239525Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-09-27T16:21:54.7259043Z http.https://github.com/.extraheader 2022-09-27T16:21:54.7285269Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-09-27T16:21:54.7304756Z http.https://github.com/.extraheader 2022-09-27T16:21:54.7335986Z Entering 'third_party/zstd' 2022-09-27T16:21:54.7356177Z http.https://github.com/.extraheader 2022-09-27T16:21:54.7591557Z Cleaning up orphan processes